Systems and methods for area-of-interest detection using slide thumbnail images

ABSTRACT

The subject disclosure provides systems and methods for determination of Area of Interest (AOI) for different types of input slides. Slide thumbnails may be assigned into one of five different types, and separate algorithms for AOI detection executed depending on the slide type. Slide types include ThinPrep (RTM) slides, tissue micro-array (TMA) slides, control HER2 slides with 4 cores, smear slides, and a generic slide. The slide type may be assigned based on a user input. Customized AOI detection operations are provided for each slide type. If the user enters an incorrect slide type, operations include detecting the incorrect input and executing the appropriate method. The result of each AOI detection operations provides as its output a soft-weighted image having zero intensity values at pixels that are detected as not belonging to tissue, and higher intensity values assigned to pixels detected as likely belonging to tissue regions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of International PatentApplication No. PCT/EP2016/051967 filed Jan. 29, 2016, which claimspriority to and the benefit of U.S. Provisional Application No.62/110,473, filed Jan. 31, 2015. Each of the above patent applicationsare hereby incorporated by reference herein in their entireties.

BACKGROUND OF THE SUBJECT DISCLOSURE Field of the Subject Disclosure

The present subject disclosure relates to digital pathology. Moreparticularly, the present subject disclosure relates to tissue detectionon glass slides containing tissue biopsies used to generate whole slidescans.

Background of the Subject Disclosure

In the field of digital pathology, biological specimens such as tissuesections, blood, cell cultures and the like may be stained with one ormore stains and analyzed by viewing or imaging the stained specimen.Observing the stained specimen, in combination with additional clinicalinformation, enables a variety of processes, including diagnosis ofdisease, prognostic and/or predictive assessment of response totreatment, and assists in development of new drugs to fight disease. Asused herein, a target or target object is a feature of the specimen thata stain identifies. A target or target object may be a protein, proteinfragment, nucleic acid, or other object of interest recognized by anantibody, a molecular probe, or a non-specific stain. Those targets thatare specifically recognized may be referred to as biomarkers in thissubject disclosure. Some stains do not specifically target a biomarker(e.g. the often used counterstain hematoxylin). While hematoxylin has afixed relationship to its target, most biomarkers can be identified witha user's choice of a stain. That is, a particular biomarker may bevisualized using a variety of stains depending on the particular needsof the assay. Subsequent to staining, the assay may be imaged forfurther analysis of the contents of the tissue specimen. Imagingincludes scanning of glass slides that contain stained tissue samples orbiopsies. The staining is important as the stain acquired bycells/microscopic structures is an indication of the pathologicalcondition of the tissue and can be used for medical diagnosis. In amanual mode, the glass slides are read by trained pathologists under themicroscope. In the digital mode, the whole slide scan is read either ona computer monitor, or it is analyzed by imaging algorithms toautomatically score/analyze the stained tissue content in the wholeslide scan. Thus, good quality scanning is absolutely essential fordigital pathology to be effectively used.

For good quality scanning, it is very important to detect tissueproperly on glass. The tissue region on the glass slide is referred toas Area of Interest (AOI). The AOI detection can be manually done fromthe thumbnail image, which is a low resolution capture of the glassslide. However, for fast, batch mode scanning, the AOI needs to beautomatically extracted from the slide thumbnail image. Unfortunately,the first step in existing methods of acquiring a thumbnail image priorto any further analysis is limited to using low resolution thumbnailcameras, and accurate and precise AOI detection is highly challenging,particularly given the variability of data with different types ofslides having different stain intensities, morphology, tissueorganization, etc. The use of a single AOI detection methodology for awide variety of slides has made solving of the AOI problem verydifficult.

SUMMARY OF THE SUBJECT DISCLOSURE

The subject disclosure solves the above-identified problems by providingsystems and computer-implemented methods for accurate determination ofAOIs for various different types of input slides. Slide thumbnail images(or thumbnails) may be assigned into one of five different types, andseparate algorithms for AOI detection may be executed depending on theslide type, with the goal being to execute operations that canefficiently and accurately compute AOI regions from whole slidethumbnail images, since a single generic solution is unable to accountfor all the data variability. Each thumbnail may be divided into one of5 types based on the tissue morphology, layout, organization of tissueon the slide thumbnail, etc. The slide types include ThinPrep (RTM)slides, also referred to as “cell line slides”, having a single discstructure (or any other similarly stained slide), tissue micro-array(TMA) slides having a grid structure, control slides having faint tissue(for a controlHER2 tissue slide, there exists a specified number ofcircular cores, usually 4), smear slides having tissue diffusedthroughout the slide, and a default or generic slide. A control slide isa slide type comprising both a tissue sample (that shall actually beexamined and analyzed) and a control tissue sample that is typicallyused for verifying histological techniques and reagent reactivity. Itcomprises a plurality of tissue regions (cores) of the tissue sample tobe actually analyzed at defined positions along a straight line.

Any slide which cannot be visually categorized to any of the first 4categories may be considered a generic slide. The slide type may beassigned based on a user input. Customized AOI detection operations areprovided for each slide type. Moreover, if the user enters an incorrectslide type, then the disclosed operations include detecting theincorrect input and executing the appropriate method. The result of eachAOI detection operations provides as its output a soft-weighted imagehaving zero intensity values at pixels that are detected as notbelonging to tissue, and higher intensity values assigned to pixelsdetected as likely belonging to tissue regions. The detected tissueregion is referred to as an Area of Interest (AOI). Optimized AOIdetection is an important step in the overall scanning process andenables subsequent steps such as focus point allocation based on thesoft weights assigned to the likely tissue regions in the output image.Higher weighted AOI regions are more likely to be assigned focus points.The described operations are superior to previous methods in terms ofprecision and recall score, and have been verified using scored computedwith ground truth AOI data.

In one exemplary embodiment, the subject disclosure provides a systemfor detecting an area-of-interest (AOI) on a thumbnail image of a tissueslide including a processor and a memory coupled to the processor forstoring computer-executable instructions that are executed by theprocessor to perform operations including receiving an input comprisinga thumbnail image and a thumbnail image type, and determining an area ofinterest (AOI) from the thumbnail image using one of a plurality of AOIdetection methods depending on the thumbnail image type, wherein upon adetermination that the thumbnail image type is incorrectly input, thedetermination of the AOI uses another of the plurality of AOI detectionmethods.

In another exemplary embodiment, the subject disclosure provides asystem for detecting an area-of-interest (AOI) on a thumbnail image of atissue slide including a processor and a memory coupled to the processorfor storing computer-executable instructions that are executed by theprocessor to perform operations including determining an area ofinterest (AOI) from a thumbnail image using one of a plurality of AOIdetection methods depending on the thumbnail image type, and outputtinga soft-weighted image depicting the detected AOI, wherein the thumbnailimage type represents one of a ThinPrep (RTM) slide, a tissue microarrayslide, a control slide, a smear slide, or a generic slide.

In yet another exemplary embodiment, the subject disclosure provides asystem for detecting an area-of-interest (AOI) on a thumbnail image of atissue slide including a processor and a memory coupled to the processorfor storing computer-executable instructions that are executed by theprocessor to perform operations including receiving an input comprisinga thumbnail image and a thumbnail image type, determining an area ofinterest (AOI) from the thumbnail image using one of a plurality of AOIdetection methods depending on the thumbnail image type, the pluralityof AOI detection methods including a ThinPrep (RTM) method, a tissuemicroarray method, a control method, a smear method, or a genericmethod, and upon determining that the thumbnail image type isincorrectly input, using the generic method.

The expressions AOI detection and tissue region detection will in thefollowing be used synonymously.

In a further aspect, the invention relates to an image analysis systemconfigured for detecting a tissue region in a digital image of a tissueslide. A tissue sample is mounted on the slide. The image analysissystem comprises a processor and a storage medium. The storage mediumcomprises a plurality of slide-type specific tissue detection routinesand a generic tissue detection routine. The image analysis system isconfigured for performing a method comprising:

-   selecting one of the slide-type specific tissue detection routines;-   checking, before and/or while the selected slide-type specific    tissue detection routine is performed, if the selected slide-type    specific tissue detection routine corresponds to the tissue slide    type of the slide depicted in the digital image;-   if yes, automatically performing the selected slide-type specific    tissue detection routine for detecting the tissue region in the    digital image-   if no, automatically performing the generic tissue detection routine    for detecting the tissue region in the digital image. The generic    tissue detection routine is also referred herein as “default” tissue    detection routine, module or algorithm.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an overview of a slide scanning process, according to anexemplary embodiment of the subject disclosure.

FIGS. 2A-2C respectively depict a system and user interfaces for AOIdetection, according to an exemplary embodiment of the subjectdisclosure.

FIGS. 3A-3B depict a method for AOI detection on ThinPrep (RTM) slidesand results of said method, according to an exemplary embodiment of thesubject disclosure.

FIGS. 4A-4B depict a method for AOI detection on tissue micro-array(TMA) slides and results of said method, according to an exemplaryembodiment of the subject disclosure.

FIGS. 5A-5D depict a method for AOI detection on control slides andresults of said method, according to an exemplary embodiment of thesubject disclosure.

FIGS. 6A-6B depict a method for AOI detection on smear slides andresults of said method, according to an exemplary embodiment of thesubject disclosure.

FIGS. 7A-7C depict a default method for AOI detection and results ofsaid method, according to an exemplary embodiment of the subjectdisclosure.

FIG. 8 depicts a histogram of a glass slide comprising a tissue sample,the histogram being used for computing multiple thresholds andcorresponding intermediate masks.

DETAILED DESCRIPTION OF THE SUBJECT DISCLOSURE

The subject disclosure provides systems, computer-implemented methodsfor accurate determination of AOIs for various different types of inputslides. The AOI detection modules described herein are vital foroptimized tissue detection on a glass slide, and are therefore anintrinsic part of an automated scanning system. Once a digital scan isobtained from a whole slide, a user may read the scanned image to comeup with a medical diagnosis (digital read), or an image analysisalgorithm can be used to automatically analyze and score the image(image analysis read). Thus, the AOI module is an enabling module, whereit enables accurate and time-efficient scanning of glass slides, andthus it enables all subsequent digital pathology applications. Asscanning is performed at the beginning of a digital pathology workflow,after acquiring a stained tissue biopsy slide, and scanning is enabledby the disclosed AOI detection. For each of the slide types, includinggeneric, a locally adaptive determination of the threshold(s) may bedetermined for the tissue, and the results merged into a “tissueprobability” image. That is, the probability of a pixel being tissue isdetermined based on both global and local constraints.

In general, once a user selects a slide type, the disclosed methodsinclude automatically performing an internal validation to check if theuser has entered a wrong slide type (corresponding to a wrongly selectedtissue-slide-type specific tissue detection algorithm), and run theproper tissue detection algorithm. For a slide marked as Control by theuser, the operations disclosed herein may determine whether the slide isan example of ControlHER2 with 4 cores lying along a straight line, andif not, a generic tissue finder module for AOI extraction may beinvoked. For a slide marked as TMA by the user, the methods maydetermine whether the slide is an example of a single rectangular gridof cores (based on training TMA slides) and if not, the disclosedmethods invoke the generic tissue finder mode for AOI extraction. ForThinPrep (RTM) images, a single core (AOI) may be captured under theassumption that the radius of the single core is known, within a rangeof variation, which has been empirically determined; and radial symmetrybased voting, performed on the gradient magnitude of a down-sampledversion of the enhanced grayscale image, is used to determine the centerof the core and its radius. Other slides stained similarly to ThinPrep(RTM) slides may be processed using this method. For a description ofradial symmetry voting to generally detect blobs or circular objects,see Parvin, Bahram, et al. “Iterative voting for inference of structuralsaliency and characterization of subcellular events.” Image Processing,IEEE Transactions on 16.3 (2007): 615-623, the disclosure of which isincorporated by reference in its entirety herein. For faint Controlimages, the operations automatically determine whether it is of type“generic” (faint slide) or Control HER2 (has 4 faint cores), and candetect the 4 cores assuming their approximate size range is known andthat their centers lie approximately in a line. The Control HER2 methoduses radial symmetry followed by Difference of Gaussian (DoG) basedfiltering so as to capture the likely locations of the core centers;then empirically determined rules are used based on how aligned the corecenter locations are, based on the distance between the core centers,and based on the angle formed by the most likely line fitted to the corecenters. For TMA images, in the condition where the slide cores arelocated in a rectangular grid, the method detects all the relevanttissue area, assuming that for the TMA cores, the individual core sizeand distance from a core to its nearest core are all very similar. Fordetecting cores, size and shape related constraints are used and oncethe cores are detected, empirically determined rules are used todetermine between genuine and outlier cores based on distanceconstraints. For smear slides, the tissue region can be reliablyidentified all through the slide assuming that the smear tissue can bespread all over the glass slide; here lower and upper thresholds havebeen computed based on luminance and color images, derived from LUVcolor space representation of the thumbnail image, where the inverse ofL is used as luminance image and square root of U-squared and V-squaredis used as the color image; hysteresis thresholding is conducted basedon these thresholds and an empirically determined area constraint. Forthe default/generic image, the tissue region is properly detected usinga range of thresholds darker than glass, and using size and distanceconstraints to detect and discard certain smaller regions. Moreover,once a certain AOI detection module is invoked, and the user can see thegenerated AOI on the thumbnail image, the user may add extra focuspoints, in the rare event that some tissue has been missed by thealgorithm. In the rare scenario where the computed AOI do notsufficiently capture all the tissue area, additional focus points can beplaced using the GUI as deemed necessary by the user, especially wherethe algorithm has failed to do so on the tissue areas.

The thumbnail image may be provided directly by an image scanner or maybe computed from an original image by the image analysis system. Forexample, an image scanner may comprise a low-resolution camera foracquiring thumbnail images and a high-resolution camera for acquiring ahigh resolution image of a tissue slide. A thumbnail image may roughlycomprise 1000×3000 pixels, whereby one pixel in a thumbnail image maycorrespond to 25.4 μm of the tissue slide. An “image analysis system”can be, for example, a digital data processing device, e.g. a computer,comprising an interface for receiving image data from a slide scanner, acamera, a network and/or a storage medium.

The embodiments described herein are merely exemplary and, although theydisclose best modes enabling persons having ordinary skill in the art toreplicate the results depicted herein, readers of this patentapplication may be able to execute variations of the operationsdisclosed herein, and the claims should be construed to encompass allvariants and equivalents of the disclosed operations, and are notlimited solely to the disclosed embodiments.

FIG. 1 depicts an overview of a slide scanning process, according to anexemplary embodiment of the subject disclosure. Generally, one or morestained tissue biopsy slides may be placed in a scanning system forimaging operations. The system may include a means 101 for capturing athumbnail image of the glass slide. This means may include a cameradesigned for the purpose. The image may be any type, including an RGBimage. Processing operations 102 may be performed on the thumbnail imageto generate a map comprising soft-weights that depict areas of interest(AOIs) for further analysis. Embodiments described herein are targetedtowards systems and computer-implemented methods for optimallyprocessing thumbnail images and accurately providing AOI maps indicatinga probability of specific or target tissue structures. The AOI maps maybe used to assign focus points 103 based on the probabilities, and foreach focus point, tiles are considered 104 around the focus points alongwith a collection of z-layers for the tile location, followed by thedetermination of the best z-layer 105, 2-dimensional interpolation 106between z-layers, and creation of an image 107 using the interpolatedlayers. The resulting image creation 107 enables further analysis stepssuch as diagnosis, prognosis, etc. This process may comprise additionalor fewer steps depending on the imaging system being used, and isintended to be construed merely to provide a context, with nolimitations intended to be imposed on any of the features describedherein.

FIG. 2A depicts a system 200 for AOI detection, according to anexemplary embodiment of the subject disclosure. System 200 may comprisehardware and software for detection of AOIs from one or more thumbnailimages. For example, system 200 may comprise imaging components 201,such as a camera mounted on a slide stage or slide tray, and/or awhole-slide scanner having a microscope and a camera. Imaging components201 generally depend on the type of image being generated. In thepresent embodiment, imaging components 201 include at least a camera forgenerating a thumbnail image of a slide. The slide may include a samplethat has been stained by means of application of a staining assaycontaining one or more different biomarkers associated with chromogenicstains for brightfield imaging. The tissue region is stained and AOIdetection involves picking up the stained tissue region. The stainedareas of interest may depict desired tissue types such as tumors,lymphatic regions in an H&E slide, etc., or hot spots of high biomarkerexpression in IHC stained slides like any tumor, immune or vesselmarkers tumor markers, immune markers, etc. The images may be scanned atany zoom level, coded in any format, and may be provided by imagingcomponents 201 to memory 210 to be processed according to logicalmodules stored thereon. Memory 210 stores a plurality of processingmodules or logical instructions that are executed by processor 220coupled to computer 225. Besides processor 220 and memory 210, computer225 may also include user input and output devices such as a keyboard,mouse, stylus, a display/touchscreen, and networking elements. Forexample, execution of processing modules within memory 210 may betriggered by user inputs, as well as by inputs provided over a networkfrom a network server or database for storage and later retrieval bycomputer 225. System 200 may comprise a whole slide image viewer tofacilitate viewing of the whole slide scans.

As described above, the modules include logic that is executed byprocessor 220. “Logic”, as used herein and throughout this disclosure,refers to any information having the form of instruction signals and/ordata that may be applied to affect the operation of a processor.Software is one example of such logic. Examples of processors arecomputer processors (processing units), microprocessors, digital signalprocessors, controllers and microcontrollers, etc. Logic may be formedfrom signals stored on a computer-readable medium such as memory 210that, in exemplary embodiments, may be a random access memory (RAM),read-only memories (ROM), erasable/electrically erasable programmableread-only memories (EPROMS/EEPROMS), flash memories, etc. Logic may alsocomprise digital and/or analog hardware circuits, for example, hardwarecircuits comprising logical AND, OR, XOR, NAND, NOR, and other logicaloperations. Logic may be formed from combinations of software andhardware. On a network, logic may be programmed on a server, or acomplex of servers. A particular logic unit is not limited to a singlelogical location on the network. Moreover, the modules need not beexecuted in any specific order. Each module may call another module whenneeded to be executed.

An input detection module may receive a thumbnail image from imagingcomponents 201, as well as a user input identifying a slide type. Thethumbnail image (or “thumbnail”) may be assigned into a category basedon the slide type input by the user, and separate algorithms 213-217 forAOI detection may be executed depending on the category. Each thumbnailmay be divided into one or more types based on the tissue morphology,layout, organization of tissue on the thumbnail. The slide types includebut are not limited to ThinPrep (RTM) slides having a single discstructure, tissue micro-array (TMA) slides having a grid structure,control slides having faint tissue with a specified number of circularcores (usually 4), smear slides having tissue diffused throughout theslide, and a default or generic slide that includes all slides that donot fit in either of the previous categories. As an example, FIG. 2Bdepicts a user interface for receiving a slide type based on a userinput. The user may enter a correct slide type on viewing the slidethumbnail based on a selection list 219. For example, the user maychoose among the following slide types: Grid (referring to TMA slideswhere the tissue cores are present in a single rectangular grid), Faint(control slides are generally faint and due to the control stains, theyare never positively stained and that makes the thumbnail image appearfaint), Round (in ThinPrep (RTM) slides, there is a single circular coreof a certain radius range present in the slide thumbnail), Disperse (forsmears, the tissue is generally spread all throughout the slide and isnot confined to a certain area like the other slide types), and Default,also referred to as generic.

Each of modules 213-217 for AOI detection are customized for each ofthese slide types. For instance, for ThinPrep (RTM) slides, the tissueAOI is typically a single disc, and hence a circle-finder in a givenradius range is an effective tool. Other slides similar to ThinPrep(RTM) slides, such as wherein the tissue sample is in a circular ornear-circular shape, may be processed using a specific module.Similarly, control HER2 slides typically have 4 cores with a knownapproximate radius range and variations in intensity from darker tofainter from bottom to top. In a smear slide, the tissue spreads andoccupies a larger part of the whole thumbnail image, and the AOI is notconcentrated in smaller, more compact shapes, as is the case for theother types. In a TMA or grid slide, the AOI is generally seen in a m×ngrid, where the grid elements can vary in size (individual TMA blobs canbe much smaller in size as compared to blobs seen in other non-TMAthumbnail images), therefore the knowledge of a grid helps putconstraints on the expected size of the constituent blobs thus helpingto weed out outlier blobs and shapes. Generic AOI detection module 217provides a catch-all operation for any slide that does notexplicitly/visually belong to these 4 slide types. The detected tissueregion for each detection module is referred to as an Area of Interest(AOI).

The result of each AOI detection operation (that may be performed by thegeneric or one of the tissue-slide-type specific tissue detectionroutines) is provided to a soft-weighted image output module 218 togenerate an output of a soft-weighted image having zero intensity valuesat pixels that are detected as not belonging to tissue, and higherintensity values assigned to pixels detected as likely belonging totissue regions. In a “soft-weighted image”, each pixel has assigned aweight, referred herein also as “soft weight”. A “soft weight” as usedherein is a weight whose value lies in a value range that comprises morethan two different values (i.e., not only “zero” and “one”). Forexample, the value range may cover all integers from 0 to 255 or may bea floating point number between 0 and 1. The weight indicates alikelihood of the pixel lying in a tissue region rather than in anon-tissue region, e.g. a glass region. The greater the value of theweight, the higher is the likelihood that the pixel lies in a tissueregion.

Soft-weighted image output module 218 is invoked to compute a softweighted image for a given luminance and color image and a binary maskthat corresponds to the AOI (i.e. the unmasked regions of the maskrepresenting tissue regions). Generally, colored pixels are more likelyto be tissue than glass; hence, pixels where the chrominance (color)component is higher are weighted higher. Darker pixels (based onluminance) are more likely to be tissue than fainter pixels. This moduleuses an RGB image, e.g. the RGB thumbnail image, and a binary mask M asits inputs, with the binary mask M provided by either of AOI detectionmodules 213-217 (i.e., provided by the generic or one of thetissue-slide-type specific tissue detection routines), and provide anoutput of a soft weighted image SW with pixel values e.g. in [0, 255].

The image is first converted from an RGB color space to an L, UV colorspace (L=luminance, U and V are color channels). Let L′=max(L)−L,wherein max(L) is the maximum luminance value observed in the converteddigital image and L is a luminance value of a currently transformedpixel; (higher L′→darker regions).

The UV color space, also referred to as chrominance color space orcomposite UV color channel is computed according to UV=sqrt(Û2+V̂2).

For the luminance image, the image analysis method identifies all pixelswhere M>0 (e.g. the regions of the image not masked by the generatedmask M), and compute lower (L′_(low): 5% value of sorted L′ values) andhigher thresholds (L′_(high): 95% value) in L′ domain for the identifiedpixels.

Similarly, compute lower (UV_(low): 5% value of sorted UV values) andhigher (UV_(high): 95% value) thresholds in UV domain. Instead of said5%, values in the range of 2-7% may also be used. Instead of said 95%,values in the range of 90-98% may also be used.

Compute WeightedL image from L′ domain, using L_(low) and L′_(high). Themapping from L′ to WeightedL upon creating the WeightedL image includessetting L′(x,y)<=L′_(low)→WeightedL(x,y)=0, and settingL_(low)<L′(x,y)<L′_(high)→WeightedL(x,y)=(L′(x,y)−L′_(low))/(L′_(high)−L′_(low)),and setting L′(x,y)>L′_(high)→WeightedL(x,y)=1. This means that a pixelof the weightedL image has a value of “0” if its original L′ value isbelow the L_(low) threshold, has a value of “1” if its original L′ valueis above the L′_(high) threshold, and has a value of(L′(x,y)−L′_(low))/(L′_(high)−L′_(low)) in all other cases. The value(L′(x,y)−L′_(low))/(L′_(high)−L′_(low)) is a normalized inverseluminance value, the normalization being performed over the unmaskedpixels.

Similarly, WeightedUV is computed from the UV image:

The image analysis system computes a WeightedUV image from UV′ domain,using UV′_(low) and UV′_(high). The mapping from UV′ to WeightedUV uponcreating the WeightedUV image includes settingUV′(x,y)<=UV′_(low)→WeightedUV(x,y)=0, and settingUV′_(low)<UV′(x,y)<UV′_(high)→Weighted UV(x,y)=(UV′(x,y)−UV′_(low))/(UV′_(high)−UV_(low)), and settingUV′(x,y)>UV′_(high)→Weighted UV (x,y)=1. This means that a pixel of theweightedUV image has a value of “0” if its original UV′ value is belowthe UV′_(low) threshold, has a value of “1” if its original UV′ value isabove the UV′_(high) threshold, and has a value of(UV′(x,y)−UV′_(low))/(UV′_(high)−UV′_(low)) in all other cases. Thevalue (UV′(x,y)−UV′_(low))/(UV′_(high)−UV′_(low)) is a normalizedinverse chrominance value, the normalization being performed over theunmasked pixels.

Then, a combined weighted image W may be created by merging theWeightedL and the WeightedUV image. For example, the weight assigned toeach pixel in the weighted image W could be computed as the average ofthe weights of said pixel in the WeightedL and the WeightedUV image.

Then, the image analysis system maps the weighted image W to a softweighted image whose pixels respectively having assigned a weight, alsoreferred to as “soft weight” between 0 and 1.

Mapping from (weighted image W, mask image M) to SW includes computing ascale factor s according to s=(255−128)/(W_(max)−W_(min)), whereinW_(max) is the maximum observed weight in the weight image W and W_(min)is the minimum observed weight in the weight image W, whereby the maxand min values in W are selectively computed for pixels of the digitalimage where M>0, and setting M(x,y)=0→SW(x,y)=0, thereforeM(x,y)>0→value=(W(x,y)−Wmin)*s+128; SW(x,y)=min(max(128,value), 255).Thereby, 255 represents the maximum possible intensity value of an RGBimage and 128 half of said value. The pixels of the soft-weighted imageSW have zero intensity at the (non-tissue) regions masked by the mask Mand have a “normalized” intensity value in the range of 128-255, wherebythe intensity value is derived from the weight of the pixel andindicates the likelihood of the pixel of being a tissue-region pixel.

Optimized AOI detection is an important step in the overall scanningprocess and enables subsequent steps such as focus point allocationbased on the soft weights assigned to the likely tissue regions in theoutput image. Higher weighted AOI regions are more likely to containfocus points.

A “focus point” as used herein is a point in a digital image that isautomatically identified and/or selected by a user because it ispredicted or assumed to be indicative of relevant biomedicalinformation. The automated or manual selection of a focus point maytrigger, in some embodiments, a scanner to automatically retrievefurther, high-resolution image data from regions surrounding said focuspoint.

However, for cases where the obtained AOI is not precise and thescanning technician or other operator of system 200 wishes to add morefocus points to obtain a better scan, such an option may be enabled viathe user interface. For example, FIG. 2C depicts an exemplary interfacefor manually adding additional focus points. The interface of FIG. 2Cmay be presented on a computer that is coupled to, for instance, ascanner for scanning tissue slides. Moreover, the described operationsare superior to previous methods in terms of precision and recall score,and have been verified using scored computed with ground truth AOI data.For example, AOI detection modules 213-217 were developed based on atraining set of 510 thumbnail images. The AOI detection modules haveused some empirically set parameters for features such as minimumconnected component size, distance of a valid tissue region from coverslip edges, possible cover slip locations, color thresholds todistinguish between valid tissue regions and dark pen colors, with theparameters being set based on the observed thumbnails as furtherdescribed herein.

Memory 210 also stores an error correction module 212. Error correctionmodule 212 may be executed upon attempting to find AOIs in the inputslide type and encountering an unexpected result. Error correctionmodule 212 includes logic to determine that the wrong slide type hasbeen input, and to select an appropriate AOI detection module regardlessof the input. For example, the TMA AOI detection module 216 may expectto see a single rectangular grid; however it may happen that there aremultiple grids in the same thumbnail. In such a case, the TMA AOIdetection may capture only a single grid pattern and miss the rest. Inthis case, generic AOI detection module 217 may be more useful todetermine all appropriate AOIs in the slide. If a user inputs a slidetype as a “faint” slide, generic AOI detection module 217 may beexecuted. For example, the user may indicate via a graphical userinterface (GUI), e.g. by pressing a button or selecting a menu item thatthe currently analyzed image is a “faint” image. A “faint” image as usedherein is an image that comprises tissue regions that are not very darkcompared to the brightness of the glass, i.e., having a low contrastbetween tissue and glass regions. In an inverted grayscale image, thebrightest parts are the most dense tissue and the darkest parts are theleast dense tissue and the completely dark (masked) parts correspond toglass. However, the term “faint” may also apply for control images, i.e.those having 4 cores. Hence, when “faint” image type is received asinput, control AOI detection module 214 may be executed, and if 4 coresare not detection, then error correction module 212 instead callsgeneric AOI detection module 217. Similarly, for some TMA images, theTMA AOI detection module 216 may retrieve only a small fraction of theprospective AOI data, thereby triggering a determination that there maybe multiple grids in the thumbnail, and therefore a determination thatgeneric AOI detection may be better than the TMA AOI detection. Theseand other alternative selections for incorrectly-input slide types aredescribed below with reference to Table 1.

TABLE 1 Operations of error correction module 212 (expected performancein parenthesis). User Input → Actual slide Control ↓ Generic TMA (HER2)ThinPrep Smear Generic (OK) Branch to (bad if it is (bad) (medium)generic a faint 4- (medium level core) performance as small TMA coresmay be missed by generic method) TMA Branch to (OK, can branch (Bad for4 (Bad) (Medium) generic if to generic if cores) Even if If neededneeded) Even if algorithm algorithm algorithm branches branches branchesto to generic to generic generic Control (OK) (Medium) (OK for 4 (Bad)(Medium) (HER2) Branch to Branch to cores) Will Even if If genericgeneric unless internally algorithm algorithm unless TMA image hasbranch to branches branches generic 4-5 cores in a generic if it togeneric to generic image vertical line is not a 4- has 4-5 core imagecores ThinPrep (Bad) (Bad) (Bad) (OK) Bad Always expects a circle whichmay not be the AOI shape in general Smear (Medium) (Bad) (Bad) (Bad)(OK) Expects a scattered AOI, compact blobs are not captured

Each row in Table 1 indicates the slide type input by a user, and eachcolumn indicates the actual-slide-type. Each cell indicates aperformance level expected from the operations (bad, medium, OK, etc.).For the smear and ThinPrep (RTM) images, default modes are not provided,i.e. there is no branching to the generic mode for either of thesecases. It should be noted that it is generally preferable to capturemore spurious tissue regions than to miss out on a potential tissueregion, since missed detections are penalized more severely than falsedetections.

FIGS. 3A-3B depict a method for AOI detection on ThinPrep(RTM) slidesand results of said method, according to an exemplary embodiment of thesubject disclosure. The method of FIG. 3 may be executed by anycombination of the modules depicted in the subsystems of FIG. 1, or anyother combination of subsystems and modules. The steps listed in thismethod need not be executed in the particular order shown. In ThinPrep(RTM) slides, regions of interest are generally circular and fainter.Therefore, contrast stretching (S331) is performed as an imageenhancement operation, enabling a single core (region of interest) tobecome more prominent. The top several rows of pixels may be discardedto remove the label (S332). For example, empirical observations showthat the top 300 rows either have label text or are blank. In otherwords, the tissue generally lies well below the top 300 rows.Subsequently, in order to enable computing both the likely center of thecircular core (S334) and its radius, the thumbnail image is downsampled(S333). For instance, the thumbnail may be downsampled thrice, each timeby a factor of 2. A pyramidal approach may be used for fasterdownsampling. The dowsampled image makes it easier/faster to compute thelikely location of the circular core (S334). Radial symmetry basedvoting may be used to find the most likely core center C (S334). Theradial symmetry operation may use a radius range of [42, 55] within thethrice down-sampled image. This corresponds to empirical observationsthat the radius of the single tissue core in the input image may liebetween 336 and 440 pixels. The AOI region is then defined (S335) as thedisk obtained at center C (disk radius=mean radius of all points whosevotes have accumulated at C). Finally, weights are assigned (S336) toAOI pixels based on luminance and color properties.

Therefore, the general assumptions used by this method are that a singlecore is present in the image, tissue content that lies in first 300pixel rows is ignored or discarded, and the radius of the single coreshould be between 336 and 440 pixels (based on empirical/trainingobservations and on the actual dimensions of slide thumbnail image).FIG. 3B depicts results of the method of FIG. 3A. Image 301 of theThinPrep (RTM) slide is delineated with a rectangle 303 marking an areaof interest that includes a tissue blob or core 304. A soft-weighted AOImap 302 (or mask) is returned by the method.

FIGS. 4A-4B depict a method for AOI detection on a tissue micro-array(TMA) slide and results of said method, according to an exemplaryembodiment of the subject disclosure. The method of FIG. 4 may beexecuted by any combination of the modules depicted in the subsystems ofFIG. 1, or any other combination of subsystems and modules. The stepslisted in this method need not be executed in the particular ordershown. Generally, this method is similar to the “generic” methoddescribed in FIG. 7A. Of the various blobs present in the thumbnail, itis determined whether or not the different blobs constitute a grid. Sizeconstraints are imposed to discard very small or very large blobs frombeing potential TMA blobs, and shape constraints used to discard highlynon-circular shapes. There may be certain cases where the TMA methodbranches off to the generic method, such as when only a small fractionof the possible AOI region as obtained from the soft weighted foregroundimage is captured in the TMA grid.

More specifically the present method starts with obtaining a grayscaleimage from RGB thumbnail image (S441), and discarding regions (S442) ofthe image that are known to be irrelevant based on training data. Forexample, black support tabs at bottom of thumbnail image may bediscarded. Cover slip margins may be detected, and regions outside thecover slip margins may be discarded. Dark regions may be detected, i.e.those having gray pixels that are <40 units in intensity (assuming 8-bitthumbnail images), and these dark regions may be discarded from beingpart of the AOI. The typical width of a coverslip is about 24 mm, butsignificantly larger or smaller coverslips can be used as well. However,embodiments of the invention assume that there exist not more than onecoverslip per slide.

In addition to these regions, margins, pen marks, dark symbols may alsobe detected and discarded. A control window may be detected, if itexists on the slide. A “control window” as used herein is a region onthe tissue slide comprising an additional tissue section with knownproperties which was subjected to the same staining and/or washingprotocol as the tissue sample to be detected and analyzed during imageanalysis. The tissue in the control window typically acts as a referenceimage for evaluating if the tissue slide washing, staining andprocessing was performed correctly. If a control window is found,regions corresponding to the control window borders may be discarded.For cleaner AOI output, regions within 100 pixels of the boundary mayalso be discarded to help avoid spurious AOI regions closer to margin,and based on the assumption that TMA cores will not lie so close to thethumbnail boundary.

A histogram is then computed (S443) for all remaining valid imageregions in the image. Based on image histogram, a range of thresholdvalues may be computed (S444) to perform adaptive thresholding. Thisoperation includes determining regions of pure glass, i.e. where theimage histogram peaks at pixel value corresponding to glass, andcomputing a range R_(A) of threshold values less than the glassintensity value. Based on the image histogram, it is further determinedif the slide is faint enough (S445) and if yes, a range R_(B) ofthreshold values is automatically created, where the range R_(B) iscloser to the glass intensity value as compared to range R_(A). A maskis created (S447) based on the threshold ranges.

According to some embodiments, the range of threshold values may becomputed as follows:

Let a pixel intensity value leading to a histogram peak bemaxHistLocation (this may correspond to glass).

Let the maximum intensity value in the grayscale image bemax_grayscale_index. This value represents a “glass_right_cutoff” value.

According to some embodiments, a range of thresholds is chosen andstored in a vector of elements, labeled, for instance,different_cutoffs, using the following operations:

Let the right gap=max_grayscale_index−maxHistLocation.

Let the glass_left_cutoff=maxHistLocation−rightgap.

Let the interval_range_gap=max(1, round(rightgap/5)).

Let min_rightgap=−rightgap.

Let max_rightgap=min(rightgap−1, round(rightgap*0.75). The value 0.75 isa predefined gap_cutoff_fraction value which typically is in the rangeof 0.5-0.95, preferentially in the range of 0.7-0.8.

Let different_cutoffs[0]=glass_left_cutoff−2*rightgap.

Letnumber_interval_terms=(max_rightgap−min_rightgap)/interval_range_gap.

The size of the vector different_cutoffs is (number_interval_terms+2).

For i=0:number_interval_terms−1:

different_cutoffs[i+1]=(glass_left_cutoff−rightgap)+interval_range_gap*i

different_cutoffs[number_interval_terms+1]=glass_left_cutoff+max_rightgap.

FIG. 8 depicts a histogram of a glass slide comprising a tissue samplethat may be used according to embodiments of the invention for a rangeof thresholds and corresponding intermediate masks. Typically, theintensity peak caused by glass regions mhl is located around 245 andtypically falls within the range of 235-250, and the mgs typically fallsin a range 0-15 units higher than mhl but is limited to a maximum of255. The basic idea is that in a typical image, the pixels correspondingto glass form a rather symmetrical peak that is just 3-10 units (out of255) wide on each side. The “rightgap” (mgs-mhl) is the observed widthof this peak on the right. “Glass_left_cutoff” is an estimate as towhere the glass ends on the left side of this peak. Below that in pixelintensity is typically tissue. However, in particular for faint tissueslides, a single threshold has been observed to not always be able tocorrectly separate tissue from glass regions. Thus, embodiments of theinvention use values identified in a histogram as depicted in FIG. 8 forautomatically extracting a range of threshold values rts ranging from apixel intensity that is very probably tissue up to a pixel intensitythat is very probably glass. In the depicted example, the thresholdrange ranges from dco[0], i.e., from a value lying two rightgaps to theleft from glco to a value that is close to mhl. For example, said valuelying close to mhl could be computed as

Dco[max-i]=different_cutoffs[number_interval_terms+1]=glass_left_cutoff+max_rightgap.It is also possible that simply mhl is chosen as the maximum thresholdvalue in the range of threshold values rts. Depending on theembodiments, the number of thresholds (number_interval_terms) containedin the threshold range its may be predefined or may be computed as aderivative of the value range spanned by the rts.

Each of the thresholds in the its threshold range is applied on thedigital image or a derivative thereof (e.g. a grayscale image) forcomputing a respective mask. The generated masks are iteratively andpair wise merged for generating intermediate masks and finally a singlemask M. Thereby, the size of pixel blobs in each mask is compared withempirically determined minimum blob areas and is checked for theoccurrence of other blobs in the neighborhood of said blob in therespective other mask.

For example, blobs in of a first mask1 and a second mask2 one of thetotality of generated masks may be identified by connected componentanalysis. In case there are unmasked pixels in mask1 within a distanceof e.g. 50 pixels of an i-th blob in mask2 (other distance thresholdstypically below 100 pixels may be used as well) and if its totalsize>=(minimum_area/2), then this blob in mask2 is considered as “trueblob” and the unmasked state of its pixels is maintained. If there areno unmasked pixels in mask1 within a distance of e.g. said 50 pixels ofpixels in i-th blob mask (i.e. the i-th blob in mask2 is relativelyisolated), the pixels of said blob in mask2 are only maintained asunmasked pixels if the blob is larger than e.g. 1.5*minimum_area. Theintermediate mask generated by merging mask1 and mask2 comprises thepixels of the blob in mask2 as unmasked pixels only in case said blob iseither sufficiently big to exceed a first empirically derived threshold(>1.5*minimum_area) or at least exceeds a second threshold that issmaller than the first threshold (>0.5*minimum_area) and which have atleast some unmasked pixels in mask1 in its neighbourhood (within 50pixels). The value of minimum_area has been empirically determined to bee.g. 150 pixels.

A soft weighted image SW (with values in the range [0,1]) is createdusing the grayscale image and using different_cutoffs. A binary mask(BWeff) is created (S447) which is ON at all pixels where the softweighted image>0. For binary mask creation (S447), for all thresholds indifferent_cutoffs, a binary mask image is created (corresponding pixelsin mask image are such where grayscale value<threshold); resulting in avector of binary mask images. Let this vector of mask images be calledmasks. Given N elements in masks, another vector of mask images iscreated, called FinalMask, which has (N−1) elements. A function tocombine masks[i] and masks[i+1] to generate FinalMask[i] may be referredto as CombineTwoMasks, and it needs as input the minimum size of a blob.A binary OR operation of all the binary images in FinalMask results inBWeff. The function which takes grayscale, different_cutoffs,minimum_blob_size, as input and returns SW, BWeff as output, is referredto as CreateWeightedImageMultipleCutoffs.

The function to combine two masks, called CombineTwoMasks, may includeoperations as follows:

Let FinalMask=CombineTwoMasks(mask1, mask2, minimum_area)

Initialization: FinalMask is made of the same size as mask1 and mask2,and all pixels in FinalMask are set to 0.

Perform connected components (CC) on mask2; let number of CC be M

Compute distance transform on mask1; let distance transformed matrix beDist

For i=1: M (looping over all M connected components), consider i-th CCin mask2 and see whether there are ON pixels in mask1 within a distanceof 50 pixels of i-th CC in mask2 (done using knowledge of Dist); if yes,then consider this region only if its total size>=(minimum_area/2) andset all pixels in FinalMask corresponding to i-th CC mask to ON. Ifthere are no ON pixels in mask1 within a distance of 50 pixels of pixelsin i-th CC mask (i.e. the i-th CC in mask2 is relatively isolated),consider the CC only when it is big enough, i.e. area of i-th CC basedmask>(1.5*minimum_area) and set all pixels in FinalMask corresponding toi-th CC mask to ON. Thus, to summarize, the role of CombineTwoMasks isto consider only those blobs in mask2 which are either sufficiently big(>1.5*minimum_area), or partly big (>0.5*minimum_area) and which have atleast some ON pixels in mask1 close enough to it (within 50 pixels). Thevalue of minimum_area has been empirically determined to be 150 pixels.

Once the binary mask BWeff is computed, soft weights are assigned to thepixels based on the grayscale intensity values, with the underlyingconcept being that pixels which are darker (lower in grayscale intensityvalues) are more likely to be tissue and hence, will receive higherweight. The soft weights also enable subsequent determination of validblobs (S449) in subsequent operations, as blobs where the sum of softweights exceeds a certain threshold may be retained and others discarded(S450). For instance, a blob that is large enough or has enough pixelswith a strong “darker” foreground may be valid, and other blobsdiscarded. To map the soft-weighted image, the input is the grayscaleimage, and effective binary mask image BWeff, the vector of thresholdvalues called different_cutoffs, and the output is the soft weightedforeground image SW. To determine ON pixels in the effective binary maskBWeff (total number of elements in the different_cutoffs vector is N):

Let norm_value=(grayscale value−different_cutoffs[0])/(different_cutoffs[N−1]−different_cutoffs[0]).

If norm_value>1, norm_value=1; else if norm_value<0, norm_value=0.

Corresponding float value in SW=1−norm_value

If corresponding grayscale value<different_cutoffs[0], correspondingfloat value in SW=1 (if pixel is dark enough, soft weight valueassigned=1).

If corresponding grayscale value>different_cutoffs[N−1], correspondingfloat value in SW=0 (if pixel is light enough, soft weight valueassigned=0).

According to embodiments, the generic tissue region detection routineand one or more of the tissue-slide-type specific tissue regiondetection routines are further configured for computing a (soft)weighted image from the digital image of the tissue sample, therebytaking a grayscale image of the tissue sample, a mask image (BWeff) anda soft weighted foreground image (SW). In SW, we consider all the pixelsbetween 0.001 and 1, using 21 uniformly spaced bins for the histogram.Depending on the distribution of data in the bins, we decide if theslide is faint enough or not. If the image is faint enough, then thedistribution of data in the higher end histogram bins will be lowenough; similarly, the distribution of data in the lower end histogrambins will be high enough.

The above mentioned steps of histogram generation, threshold extraction,Mask generation, mask fusion and/or (soft) weight computation may beimplemented by a sub-routine of the generic and/or one of thetissue-slide-type-specific tissue detection routines.

Once a binary mask is generated (S447) per the steps described above,connected components detection (S448) may be performed to see if eachconnected component is a possible TMA core. Detected connectedcomponents are subject to rules regarding valid size, shape, anddistance (S449), as follows:

Width of the connected component (CC)<(cols/6): (width of thumbnailimage, below label,=cols)

Height of the CC<(rows/8): (height of thumbnail image, belowlabel,=rows)

Area of CC>55 pixels (small junk blobs may be discarded)

Eccentricity of CC mask>0.62 (TMA cores are assumed to be circular)

After finding possible TMA cores, a distribution of sizes of CC isdetermined to decide which TMA cores are proper enough:

Let MedianArea=median of all CC retained after the previous step ofidentifying valid CC.

Retain those CC whose area >=MedianArea/2 and area <=2*MedianArea.

Then, the inter-blob distance of all the valid CC is checked, and thefollowing rule is used to decide whether a blob is to be retained ordiscarded. The terms used in the rule may include:

NN_distvalues[i]=distance between i-th blob center to center of nearestTMA blob.

radius_current_blob_plus_NN_blob[i]=radius of i-th TMA blob+radius of(blob nearest to i-th TMA blob).

DistMedian=median of (NN_distvalues).

MADvalue=mean absolute deviation of (NN_dist_values).

Sdeff=max(50, 2*MAD).

The rule subject to blobs may be represented as:NN_dist_values[i]<=1.3*(DistMedian+max(Sdeff,radius_current_blob_plus_NN_blob[i])=>for i-th blob to be a valid TMAblob. When TMA cores are considered, the distance between any twonearest cores is generally similar in a TMA, therefore all theseassumptions are used in the distance based rules.

Therefore, the TMA algorithm of FIG. 4A is based on the assumptions thatthe valid TMA cores are small enough, the valid TMA cores are largerthan a certain threshold, so as to discard small junk/dirt/dust basedblobs, each valid TMA core is circular enough and so eccentricity shouldbe closer to 1, and all valid TMA cores should be close enough and thecore-to-nearest-core distance is similar enough such that when thisdistance constraint is violated, these cores are discarded from the AOI.It may be further assumed that there is no tissue region outside thedetected coverslips. For the module that detects dark regions in theimage, the assumption is that the pen-marks, symbols, are darker thantissue regions. Therefore, a grayscale value of 40 may be used todiscard dark regions. FIG. 4B shows the results of the method of FIG.4A. Image 401 of the TMA slide is delineated with a rectangle 403marking an area of interest that includes an array of tissue samples404. A soft-weighted AOI map 402 (or mask) is returned by the method.

FIGS. 5A-5B depict a method for AOI detection on controls slides andresults thereof, according to an exemplary embodiment of the subjectdisclosure. The method of FIG. 5 may be executed by any combination ofthe modules depicted in the subsystems of FIG. 1, or any othercombination of subsystems and modules. The steps listed in this methodneed not be executed in the particular order shown. This method isgenerally used to detect 4 tissue section cores (of the tissue undertest, not of the control tissue in a control window of the slide) on acontrol slide using radial symmetry operations on an enhanced version ofthe grayscale image with the motivation being that the votes willaccumulate at the centers of the 4 cores. A multi-scale difference ofGaussians (DOG) operation is used on the votes image to detect the corecenters. The radial symmetry voting helps in accumulation of votes, andwhen multi-scale DoG is used on the votes image, it helps to betterlocalize the core centers. Then, segmentation techniques may be appliedto the regions close to where the core center is found, as the blob isdarker than its immediate background and therefore thresholding helps tosegment out the cores. Although the method is directed towards “control”slides, it may work for both generic faint slides and also for controlHER2 (4-core) images. This method may automatically branch off to thegeneric method if the thumbnail does not represent a 4-core ControlHER2image.

Specifically, the method obtains a grayscale image (S551) of the RGBthumbnail, and performs radial symmetry voting (S552) and DOG (S553) onthe image to obtain the cores in the image (S554). If the image appearsto be a generic tissue slide, with no specific organization, andnon-brown in color (control slides are those where control colors arepresent and control stained tissue is not positively stained), it isdetermined to not be a 4-core control slide, and the method branches togeneric (S555). The determination is accomplished by computing mean andmaximum votes from radial symmetry voting (S552) and DOG (S553) for allthe detected cores and is based on (mean, max) of (votes, DOG) and theassumption that 4-core slides have higher values of (mean, max) for(votes, DOG). The grayscale image may be downsampled twice to speed upthe voting process. For voting, a radius range of [13, 22] pixels may beused for twice down-sampled images. DOG (S553) is performed on theresulting vote matrix, and the top 10 DOG peaks greater than a thresholdare used as prospective core center locations. For example based on allthe detected core centers, a mean and maximum of DOG for all cores and amean and maximum of votes for all cores may be computed, withmean_DOG_all_cells_cutoff set to 1.71, max_DOG_all_cells_cutoff set to2, mean_votes_all_cells_cutoff set to 1.6, max_votes_all_cells_cutoffset to 1.85, and then determine if mean/max of DOG/votes exceeds thesecutoffs. If out of 4 conditions, this condition is met for 2 or morecases, it may be decided that the image represents a control slide with4 cores.

Given prospective core centers, an Otsu thresholding-based segmentationoperation (S556) is performed to segment out the foreground blobs, basedon the fact that, in general, the core is darker than its immediatebackground, which makes it possible to extract out the core based onintensity. Once the segmentation (S556) is performed, blobs that do notmatch (S557) the radius range of [13, 22] may be discarded (S558). Blobsare expected to be circular, so blobs where the computed circularityfeature<min_circularity (0.21) are discarded, with circularity beingdefined as (4*PI*area/(perimeter*perimeter)). Circularity is low fornon-circular elongated shapes and is close to 1 for approximatelycircular shapes.

Of all the remaining cores that are in the valid radius range [13,22],any 4 blobs that lie along a straight line are detected (S559) alongwith a nearest core distance assumed to be in the range of [38, 68]pixels, i.e. the range between two nearest core centers. To determinewhether some blobs are along the same straight line (S559), a blob maybe regarded as part of a line where the distance from the blob center isless than a certain distance constraint (23 pixels). If after usingthese size and distance constraints, fewer than 4 cores are found(S560), detection of additional cores (S561) is attempted to eventuallyfind 4 cores in total, as further described with respect to FIGS. 5C-5D.

This method generally assumes that 4-core and non-4-core control imagescan be separated properly if the non-4-core images do not have 5 or moresmaller cores. Some 5-core images may be analyzed as generic images. Theradius of each core is assumed to be in the range [52, 88] pixels in thethumbnail image. To segment each core, it is assumed to be darker thanits immediate background. This makes intensity based thresholdingpossible on the grayscale version of the thumbnail image. The distancebetween two nearest cores (for a 4-core Control HER2 thumbnail image) isassumed to be in the range [152, 272] pixels. FIG. 5B shows results ofthis method. Thumbnail image 501 of the control slide is delineated witha rectangle 503 marking a tissue slide region that includes cores 504. Asoft-weighted AOI map 502 (or mask) is returned by the method. FIGS.5C-5D depict means for finding more cores (S561). For example, for everydetected core 504, the prospective cores are assumed to be 50 pixelsaway based on arrow 506. A better configuration is one which has morenumber of valid blobs (i.e. one that matches the size and circularityconstraints) and, if both the configurations have the same number ofvalid blobs, then the circularity feature, averaged over all the validcores, is compared to settle ties and select the best of the threeoptions. In FIG. 5C, only 2 blobs are found, so the method selectsbetween either of 3 alternate configurations. In FIG. 5D, only 3 blobsare found, so the method selects between the best of 2 alternateconfigurations.

FIGS. 6A-6B depict a method for AOI detection on smear slides (e.g.slides comprising a blood sample), according to an exemplary embodimentof the subject disclosure. The method of FIG. 6 may be executed by anycombination of the modules depicted in the subsystems of FIG. 1, or anyother combination of subsystems and modules. The steps listed in thismethod need not be executed in the particular order shown. Generally,the smear method considers two image channels, computed from LUV imagewhich is derived from the RGB image, L and sqrt(Û2+V̂2), where L is theluminance channel and U,V are the chrominance channels. The median andstandard deviation are computed for both the L and sqrt(Û2+V̂2) images.Based on lower and higher thresholds computed on median and standarddeviation, hysteresis thresholding operations are performed to returnseparate AOI masks for L and sqt(Û2+V̂2) domains, and then the masks arecombined. Post-processing operations may be performed to get rid ofspurious regions especially close to the bottom of the slide thumbnail.

Specifically, a grayscale image may be obtained (S661) from the RGBthumbnail image. A dark mask is generated (S662) that corresponds topixels having grayscale values <40. The RGB image is then converted(S663) to the L, UV sub-channels (L=luminance, U,V: chrominance). TheL-channel is also referred to as luminance image. The U- and V-channelsare color channels. Thresholds are computed (S664) for bothsub-channels, i.e., for the luminance image and for the chrominanceimage.

For the L channel, an inverted channel L′ is computed according toL′=max(L)−L, whereby max(L) relates to the maximum luminance valueobserved in the L-channel image and L is the luminance value of a pixel.Then, a median (MedianL) and mean absolute deviation (MADL) are computedfor the L′ channel.

The chrominance image UV is computed according to UV=sqrt(Û2+V̂2).

For the L′ channel, a lower threshold thL_(low) is computed according tothL_(low)=MedianL+MADL, and a higher threshold thL_(high) is computedaccording to thL_(high)=MedianL+2*MADL.

A similar method is used to compute the thresholds on the UV channel.

The chrominance image UV is computed according to UV=sqrt(Û2+V̂2).

An inverted channel UV′ is computed according to UV′=max(UV)−UV, wherebymax(UV) relates to the maximum luminance value observed in the UV− imageand UV is the chrominance value of a pixel in said image. Then, a median(MedianUV) and mean absolute deviation (MADUV) are computed for the UV′channel.

For the chrominance UV channel, a lower threshold thUV_(low) is computedaccording to thUV_(low)=MedianUV+MADUV, and a higher thresholdthUV_(high) is computed according to thUV_(high)=MedianUV+2*MADUV.

Then, hysteresis thresholding (S665) is performed on both channels (L′,UV′). On the L′ channel, hysteresis thresholding using thL_(low) andthL_(high) and using an area constraint of e.g. 150 pixels is used toobtain a binary mask in L′ domain (M_(L′)).

Hysteresis thresholding on the UV channel uses an area constraint ofe.g. 150 pixels to obtain a binary mask in UV domain (M_(U,V)).

“Using an area constraint” in hysteresis thresholding means that anypixel blob detected by the hysteresis thresholding (as being a set ofadjacent pixels having a value that meets the lower and the upperthreshold) is only maintained as an “unmasked” pixel blob in thegenerated mask if said blob is at least as large as the area constraint.The size of the area constraint is typically in the range of 130-170pixels, e.g. 150 pixels.

The final effective binary mask M is obtained by a binary OR combination(S666) of M_(L′) and M_(UV) (M(x,y)=1 if ML′(x,y)=1 or MUV(x,y)=1).

Performing the hysteresis thresholding on the two different masksgenerated from a luminance and from a chrominance image separately andthen recombining the thresholded masks may increase accuracy oftissue-vs-glass region detection in particular for smear slides. This isbecause the tissue regions on the glass slide typically consist of cellsspread out over the whole slide. Often said cells have low contrastrelative to the glass slide. Nevertheless, the color componentrepresented in the chrominance image may act as a clear indicator that aparticular slide region is covered by tissue.

The resulting combination may be soft-weighted as described herein togenerate the final mask. Moreover, the region in between the supporttabs, at bottom of thumbnail image, is analyzed (S667) to determine ifany tissue region in between the support tabs that is contiguous withtissue AOI region which has already been detected needs to be includedin the mask.

This method generally assumes that any cover slip detection may resultin discarding smear tissue, so tissue regions outside the cover slip arediscarded automatically. The benefit here is that if tissue does notextend to the sides/edges, the coverslip margins are ignored. Moreover,since smear tissue can lie in between support tabs, it is assumed to begenuine tissue, particularly if it is connected with the tissue alreadydetected in the region outside support tabs. For this method, after theAOI is found, smaller connected components are not discarded based onthe assumption that smear tissue may not consist of concrete blobs butthere can be pieces of tissue lying near the prominent smear portionwhich can be small enough in area. Since missing tissue regions is morerisky than capturing spurious regions, a conservative approach includesretaining the smaller connected components in the AOI. FIG. 6B showsresults of this method. Thumbnail image 601 of the smear slide isdelineated with a rectangle 603 marking an area of interest thatincludes a tissue smear 604. A soft-weighted AOI map 602 (or mask) isreturned by the method.

FIGS. 7A-7C depict AOI detection for generic or mis-identified slides,according to exemplary embodiments of the subject disclosure. The methodof FIG. 7 may be executed by any combination of the modules depicted inthe subsystems of FIG. 1, or any other combination of subsystems andmodules. The steps listed in this method need not be executed in theparticular order shown. Generally, this method uses a variety ofthresholds to come up with a soft weighted image where higherprobability of finding tissue regions is expected, and to discard faintimage regions from a faint artifact based on the assumption that faintregions close to darker tissue regions are more likely to be tissueregions than faint regions which are more distant from genuine tissueregions.

The method begins by obtaining a grayscale image (S771) from the RGBthumbnail image, and discarding regions (S772) that are known not to beof interest. For example, black support tabs at bottom of thumbnailimage are discarded, as well as regions outside a detected margin of acover slip. Dark regions having grayscale values <40) are discarded frombeing part of AOI, as well as any regions outside a margin or border ofa control window (if any is detected). An image histogram is computed(S773) for all image regions not discarded previously, based on which apeak value for glass is obtained and used to compute the range ofthreshold values over which adaptive thresholding is performed to detectprospective tissue regions. These operations S773-S776 are analogous tothose performed by the TMA method described with reference to FIG. 4A.However, in this case, if the image is a faint image, the mostsignificant blob is retained (S777), i.e. where the sum of the softweights of all the pixels belonging to the blob is at a maximum.Further, smaller blobs close to this most significant blob are retainedwhen the soft weights are high enough and are close enough to the mostsignificant blob. Connected components are detected (S778), and size,distance, and soft weights constraints applied (S779-S780) to retain anddiscard blobs. The result is an AOI mask. Further, pixels in the100-pixel-wide border region—when there are proper tissue regions in theinterior part which may touch tissue regions in the zeroed out part, areconsidered as part of the AOI mask. Small blobs are discarded (S780)based on size, distance constraint, proximity to dark support tabs atbottom, soft weight constraints, etc.

This default or generic method assumes that there is no tissue regionoutside the detected cover slips, because if the coverslip isincorrectly detected (e.g. when there are line-like patterns in thetissue region), it runs the risk of zeroing out all regions outside thefalse cover slip. Further, for the module that detects dark regions inthe image, the intuition is that the pen-marks, symbols, are darker thantissue regions. The method further assumes a grayscale value of 40 todiscard dark regions. Blobs are considered small if they are <150pixels, and discarded if they are not within a certain distance (<105pixels) from any other significant non-small blob. Pixels, with softweight >0.05 and within 75 pixels of the most significant blob, areregarded as part of the AOI.

As described above, these AOI detection algorithms have been trainedusing ground truths established from 510 training slides that containedat least 30 slides from each of all the 5 categories. The algorithm wasfurther tested on a test set of 297 slide thumbnails, different from thetraining set. Based on the ground truth AOI data, precision and recallscores were computed and it was observed that for all the 5 slide types,the precision-recall based score obtained using the these methods wasmarkedly superior than prior art methods. These improvements may beattributed to several features disclosed herein, including but notlimited to the adaptive thresholding (which uses a range of thresholdvalues versus one single statistically found threshold as was generallydone in the prior art), or a lower and higher threshold as in the caseof hysteresis thresholding, connected component detection performed onthe binary mask to discard smaller blobs, and using a range of thresholdvalues to remove dependence on a single chosen threshold, as well asconsidering the distance between smaller (yet significant) blobs andbigger blobs (which is significant as small variations in intensity cansplit larger blobs and smaller yet important blobs can get discarded byprior art methods). The operations disclosed herein may be ported into ahardware graphics processing unit (GPU), enabling a multi-threadedparallel implementation.

In a further aspect, the invention relates to an image analysis systemconfigured for detecting a tissue region in a digital image of a tissueslide. A tissue sample is mounted on the slide. The image analysissystem comprises a processor and a storage medium. The storage mediumcomprises a plurality of slide-type specific tissue detection routinesand a generic tissue detection routine. The image analysis system isconfigured for performing a method comprising:

-   selecting one of the slide-type specific tissue detection routines;-   checking, before and/or while the selected slide-type specific    tissue detection routine is performed, if the selected slide-type    specific tissue detection routine corresponds to the tissue slide    type of the slide depicted in the digital image;-   if yes, automatically performing the selected slide-type specific    tissue detection routine for detecting the tissue region in the    digital image-   if no, automatically performing the generic tissue detection routine    for detecting the tissue region in the digital image.

Said features may be advantageous as a fully automated image analysissystem is provided capable of identifying tissues in a plurality ofdifferent slide types. In case a wrong slide type and correspondingtissue analysis method is selected by a user or automatically, thesystem may automatically determine a wrong selection and will perform adefault tissue detection routine that has been shown to work well forall or almost all types of tissue slides, also. The embodimentsdescribed in the following can freely be combined with any of the abovedescribed embodiments.

According to embodiments, the digital image is a thumbnail image. Thismay be beneficial as thumbnails have a lower resolution compared to“full sized” tissue slide images and may thus be processed moreefficiently.

According to embodiments, the image analysis system comprises a userinterface. The selection of the slide-type specific tissue detectionroutine comprises receiving a user's selection of a tissue slide typevia the user interface; and selecting the one of the slide-type specifictissue detection routines assigned the tissue slide type selected by theuser.

According to embodiments, the generic tissue detection routine and oneor more of the slide-type specific tissue detection routines (the TMAroutine) comprise a sub-routine for generating a binary mask M thatselectively masks non-tissue regions. The sub-routine comprises:

-   computing a histogram of a grayscale version of the digital image;-   extracting a plurality of intensity thresholds from the histogram;-   applying the plurality of intensity thresholds on the digital image    for generating a plurality of intermediate masks from the digital    image;-   generating the binary mask M by combining all the intermediate    masks.

According to embodiments, the extraction of the plurality of intensitythresholds from the histogram comprises:

-   identifying a max-grayscale-index (mgs), the max-grayscale-index    being the maximum grayscale intensity value observed in the    histogram;-   identifying a max-histogram-location (mhl), the    max-histogram-location being the grayscale intensity value having    the highest occurrence frequency in the histogram;-   computing a rightgap by subtracting the maximum-histogram-location    (mhl)from the max-grayscale-index (mgs);-   computing a glass-left-cutoff according to    glass-left-cutoff=max-histogram-location(mhl)-rightgap;-   performing the extraction of the plurality of thresholds such that    the lowest one (dco[0]) of the thresholds is an intensity value    equal to or larger than glass_left_cutoff−rightgap and such that the    highest one of the thresholds is equal to or smaller than    glass_left_cutoff+rightgap. For example, the lowest threshold can be    an intensity value equal to or larger than    glass_left_cutoff−rightgap +1 and the highest one of the thresholds    can be equal to or smaller than glass_left_cutoff+rightgap −1.

Said features may be advantageous as the thresholds are histogram basedand therefore dynamically adapt to the faintness of the tissue- andnon-tissue regions of the slide that may depend on many differentfactors such as light source, staining intensity, tissue distribution,camera settings and the like. Thus, a better comparability of images ofdifferent tissue slides and a better overall accuracy of the tissuedetection may be achieved.

According to embodiments, the extraction of the plurality of intensitythresholds from the histogram comprises:

-   computing an interval-range-gap(irg) according to irg=max(1,    round(rightgap/constant)), wherein constant is a predefined    numerical value between 2 and 20, preferentially between 3 and 8;-   computing a max_rightgap value according to    max_rightgap=min(rightgap−1, round(rightgap*gap_cutoff_fraction),    wherein the gap_cutoff_fraction is a predefined value in the range    of 0.5-0.95, preferentially 0.7-0.8;-   computing a number_interval_terms according to    number_interval_terms=((max_rightgap+rightgap)/interval_range_gap);    and-   creating the plurality of threshold such that their number is    identical to the computed number_interval_terms.

Said features may be advantageous because the number of generatedthresholds are histogram based and therefore are dynamically adapted tothe faintness and intensity distribution of the tissue-and non-tissueregions of the slide that may depend on many different factors such aslight source, staining intensity, tissue distribution, camera settingsand the like. It has been observed that in case the glass-relatedintensity peak is very narrow, a smaller number of thresholds may besufficient for accurately separating tissue regions from glass-regions.In case the glass-related intensity peak is very broad andcorrespondingly the rightgap is large, it may be preferably to compute alarger number of thresholds and corresponding masks

According to embodiments, the application of the plurality of intensitythresholds on the digital image for generating a plurality ofintermediate masks comprises:

-   -   a) creating a first intermediate mask by masking all pixels in        the digital image whose grayscale value is below a first one of        the plurality of thresholds;    -   b) creating a second intermediate mask by masking all pixels in        the digital image whose grayscale value is below a second one of        the plurality of thresholds; the second threshold may be, for        example, the next highest ones of the plurality of thresholds;    -   c) performing a connected-component-analysis on the unmasked        pixels of the first mask for identifying first blobs;    -   d) performing a connected-component-analysis on the unmasked        pixels of the second mask for identifying second blobs;    -   e) merging the first and second mask into a merged mask, the        merging comprising:        -   masking all first blobs whose size is below an            absolute-minimum-area and masking all first blobs whose size            is above the absolute-minimum-area but below a            conditional-minimum-area and which lack a first blob in a            predefined neighborhood area around said first blob;        -   after having performed the masking of one or more of the            first blobs, unifying the unmasked regions of the first and            second mask to generate the unmasked regions of the merged            mask, all other regions of the merged mask being masked            regions;        -   using the merged mask as a new first intermediate mask,            selecting a third one of the threshold for computing a new            second intermediate mask from the third threshold, repeating            steps c)-e) until each of the plurality of thresholds was            selected, and outputting the finally generated merged mask            as the merged binary mask M.

The multi-threshold based multi-mask merging as described above may beadvantageous as different thresholds are capable of catching dots orblobs of tissue of very different intensity levels. In case one maskmisses a blob, said blob may be caught by a mask based on a less strictthreshold. Further, the way the masks are combined prevents that themasks are polluted by noise, because context information of twodifferent masks is evaluated. An unmasked pixel blob of a mask is onlymaintained in the mask merging process if said blob is either very largeor is in spatial proximity to a blob detected in the other mask. Such afinding will typically be made if a tissue region is analyzed, but notif an artifact in a glass region is analyzed. The merging of aluminance-based mask and of a chrominance-based mask is performed in thesame way as described above, whereby the luminance mask may for examplebe used as the first mask, the chrominance mask may be used as thesecond mask and a connected component analysis is performed fordetecting first and second blobs in unmasked regions of the luminanceand chrominance channel, respectively.

According to embodiments, the absolute-minimum-area is in the range of100%-200% of an expected size of a blob representing a tissue cell, theconditional-minimum-area being in the range of 30%-70% of an expectedsize of a blob representing a tissue cell. The predefined neighborhoodarea around a blob may be defined by a pixel belt around a blob having awidth of about 1-3 diameters of a typical tissue cell, e.g. a width of50 pixels.

According to embodiments, the slide-type specific tissue detectionroutines comprise a TMA-slide-routine. The TMA-slide routine isconfigured for detecting the tissue region in a tissue micro array (TMA)slide. The TMA-slide routine is configured for:

-   generating a binary mask M by performing the sub-routine according    to embodiments described above;-   applying the binary mask (M) on the digital image and selectively    analyzing unmasked regions in the digital image for detecting a grid    of tissue regions. A “core” as used herein is an individual tissue    sample or part thereof contained on a slide that is separated from    other tissue regions on the slide (other “cores”) by a predefined    minimum distance, e.g. at least two or more mm.

Embodiments of the invention make use of a plurality of thresholds forintermediate mask generation and then combine the masks for generating afinal mask. The image analysis system uses the final mask foridentifying tissue regions. Applying multiple different thresholds formask generation and then combining the masks via a smart mask markingalgorithm has been observed to provide higher accuracy intissue-vs-glass detection. The way the multiple thresholds are generatedand the respective masks are fused may slightly vary for the differenttissue slide types to further increase the accuracy of the tissuedetection algorithm.

According to embodiments, the selected slide-type specific tissuedetection routine is a TMA-slide routine and the image analysis systemis further configured for:

-   if the grid of tissue regions is not detected in the unmasked    regions of the digital image, determining that the selected    TMA-slide tissue detection routine does not correspond to the tissue    slide type of the tissue slide depicted in the digital image; and-   terminating execution of the TMA-slide tissue detection routine and    starting execution of the generic tissue detection routine.

According to embodiments, the slide-type specific tissue detectionroutines comprise a control-slide-routine. The control-slide routine isconfigured for detecting the tissue region in a control slide. A controlslide is a slide comprising a control tissue in addition to the tissuethat is to be analyzed. The control tissue and the tissue to be analyzedare stained with a control stain, i.e., a biomarker-unspecific stain,but not with a biomarker-specific stain. Due to the control stains,control tissue sections and the tissue sections to be actually analyzedare never positively stained and are typically fainter than experimentaltest slides. The control-slide routine is configured for:

-   generating a binary mask M by performing the sub-routine of any one    of the above described embodiments for binary mask generation.-   applying the binary mask M on the digital image; and-   selectively analyzing unmasked regions in the digital image for    detecting a number of tissue regions positioned in a straight line.

According to embodiments, the selected slide-type specific tissuedetection routine is a control-slide routine. In case the number oftissue regions in a straight line are not detected in the unmaskedregions of the digital image, the image analysis system determines thatthe selected control-slide tissue detection routine does not correspondto the tissue slide type of the tissue slide depicted in the digitalimage and terminates execution of the control-slide tissue detectionroutine and starting execution of the generic tissue detection routine.

According to embodiments, the slide-type specific tissue detectionroutines comprise a cell-line-slide-routine. The cell-line-slide-routineis configured for detecting the tissue region in a cell-line-slide. Acell-line-slide is a tissue slide comprising a single, disc-shape tissuesample, the cell-line-slide routine being configured for:

-   computing a grayscale image as a derivative of the digital image;    according to some embodiments, the grayscale image is an “enhanced”    image, e.g. by means of contrast stretching. The (enhanced)    grayscale image is, according to embodiments, down-sampled (stored    with reduced resolution) multiple times (e.g. trice with factor    of 2) before the gradient magnitude image is computed.-   computing a gradient magnitude image as a derivative of the    grayscale image;-   computing a histogram of the digital image;-   extracting a plurality of intensity thresholds from the histogram;-   applying a plurality of intensity threshold values on the digital    image for generating a plurality of intermediate masks from the    digital image;-   generating a binary mask (M) by combining all the intermediate    masks;-   applying the binary mask (M) on the digital image and the gradient    magnitude image; and-   performing radial symmetry based voting selectively on the unmasked    region in the gradient magnitude image for detecting a center and a    radius of a single tissue region.

According to embodiments, the selected slide-type specific tissuedetection routine is a cell-line-slide routine and the image analysissystem is further configured for:

-   if the single tissue region was not detected or if the detected    radius or the detected location of the center deviates from a radius    or center location expected for the cell-line-slide type,    determining that the selected cell-line-slide tissue detection    routine does not correspond to the tissue slide type of the tissue    slide depicted in the digital image; and-   terminating execution of the cell-line-slide tissue detection    routine and starting execution of the generic tissue detection    routine.

According to embodiments, the slide-type specific tissue detectionroutines comprises a smear-slide-routine. The smear-slide routine isconfigured for detecting the tissue region in a smear slide. A smearslide is a slide type where on which the tissue region is diffusedthroughout the slide. The smear-slide routine is configured for:

-   computing a luminance image as a derivative of the digital image,    the luminance image being an L-channel image of a LUV color space    representation of the digital image;-   computing a chrominance image as a derivative from the digital    image, the chrominance image being a derivative of the U,V-channels    of the LUV color space representation of the digital image;-   performing hysteresis thresholding on an inverted version of the    luminance image for generating a luminance-based mask;-   performing hysteresis thresholding on an inverted version of the    chrominance image for generating a chrominance-based mask;-   combining the luminance-based mask and the chrominance-based mask to    obtain a binary mask (M);-   applying the binary mask on the digital image; and-   selectively analyzing unmasked regions in the digital image for    detecting the diffused tissue region.

According to embodiments, performing the hysteresis thresholdingcomprises:

-   determining, for each set of adjacent pixels whose pixel value meets    a lower and an upper threshold applied during the hysteresis    thresholding, if the set of pixels covers at least an empirically    determined area; for example, the empirically determined area can    correspond to a typical number of pixels of a tissue cell depicted    in the digital image at the given resolution;-   if yes (if the pixel set covers at least said area), generating the    binary mask such that said set of pixels is unmasked; considering    said pixels as unmasked pixels implies that said pixels are    considered as tissue regions while the masked pixels are considered    as non-tissue regions, e.g. glass regions;-   if no, generating the binary mask such that said set of pixels is    masked.

Comparing the adjacent sets of pixels may have the advantage that smallspeckles and noise that is smaller than a typical cell is filtered outby the generated mask.

According to embodiments, the tissue slide is a glass slide and thetissue detection routines are configured for generating masks thatselectively mask glass regions, wherein tissue regions are unmaskedregions in said masks.

According to embodiments, the sub-routine further comprising using sizeand distance constraints to detect and discard smaller regions fromintermediate masks when combining the intermediate masks to generate thebinary mask.

According to embodiments, the generic and one or more of thetissue-slide-type-specific tissue detection routines are furtherconfigured for generating a weighted image from the digital image, eachpixel in the weighted image having assigned a weight in a value rangecomprising more than two different values. The weight indicates alikelihood that said pixel belongs to a tissue region as compared tobelonging to glass. This may be advantageous, as a mask merely generatesa binary image but the likelihood of being a tissue region rather aglass region may be between zero and one. For some pixels it may hardlybe possible to unambiguously decide if they are tissue pixels or not. Byassigning weights that may have any of a plurality of possible values,it is possible to display to a user some prospective tissue regionshaving a high likelihood of being truly a tissue pixels and someprospective tissue regions having a lower likelihood of being a tissueregion (but nevertheless being unmasked and thus being considered as atissue region).

According to embodiments, the generating of the weighted image uses thedigital image as well as a binary mask M as an input. The soft-weightedimage is zero valued at all pixels except the pixels which are non-zeroinside the binary mask, and these non-zero pixels get assigned higher orlower values depending on whether they are more or less likely to belongto tissue as compared to belonging to glass.

According to embodiments, the generation of the weighted imagecomprising:

-   computing a luminance image as a derivative of the digital image,    the luminance image being an L-channel image of a LUV color space    representation of the digital image, or accessing a luminance image    having been computed already for generating a binary mask (M);-   computing a chrominance image as a derivative from the digital    image, the chrominance image being a derivative of the U,V-channels    of the LUV color space representation of the digital image, or    accessing a chrominance image having been computed already for    generating the binary mask M;-   masking the luminance and/or the chrominance with a binary mask (M),    the binary mask selectively masking non-tissue regions;-   generating a weighted image SW as a derivative of the luminance    and/or the chrominance image, wherein each pixel in the weighted    image has assigned a value that positively correlates with the    chrominance of a corresponding pixel in the luminance image and/or    that positively correlate with the darkness of a corresponding pixel    in the luminance image.

Generally, colored pixels (whose color information is contained in thechrominance image) and darker pixels (luminance image) are more likelyto be tissue than glass; hence, pixels with a high chrominance (color)component and/or darker pixels (according to luminance) are weightedhigher.

The computation of the chrominance UV and luminance L image areperformed by the image analysis system as already described forembodiments of the invention.

According to embodiments, the image analysis system comprises agraphical user interface configured for displaying a selectable GUIelement. The GUI element enables a user to select one of the slide-typespecific or the generic tissue detection routines. The slide-typespecific tissue detection routine is one of a TMA-slide-routine, acontrol-slide-routine, cell-line-slide-routine, and asmear-slide-routine. In addition or alternatively, the image analysissystem displays the detected tissue regions, wherein the color and/orbrightness of the pixels of the displayed tissue regions depend on theweights [soft-weights] assigned to each of the tissue region pixels. Inaddition or alternatively, the GUI enables a user to add extra focuspoints on regions of the displayed (soft-weighted) tissue regions, theadding of the extra focus points triggering a scanner to scan regionsindicated by the focus points in more detail.

In a further aspect, the invention relates to an image analysis methodfor detecting a tissue region in a digital image of a tissue slide. Atissue sample is mounted on the slide. The method is implemented in animage analysis system comprising a processor and a storage medium. Themethod comprises:

-   selecting one of the slide-type specific tissue detection routines;-   checking, before and/or while the selected slide-type specific    tissue detection routine is performed, if the selected slide-type    specific tissue detection routine corresponds to the tissue slide    type of the slide depicted in the digital image;-   if yes, automatically performing the selected slide-type specific    tissue detection routine for detecting the tissue region in the    digital image-   if no, automatically performing the generic tissue detection routine    for detecting the tissue region in the digital image.

In a further aspect, the invention relates to a storage mediumcomprising instructions interpretable by a processor of an imageanalysis system, the instructions when executed by the processor causethe image analysis system to perform the method according to an imageprocessing method implemented in an image analysis system according toany one of the embodiments described herein.

Computers typically include known components, such as a processor, anoperating system, system memory, memory storage devices, input-outputcontrollers, input-output devices, and display devices. It will also beunderstood by those of ordinary skill in the relevant art that there aremany possible configurations and components of a computer and may alsoinclude cache memory, a data backup unit, and many other devices.Examples of input devices include a keyboard, a cursor control devices(e.g., a mouse), a microphone, a scanner, and so forth. Examples ofoutput devices include a display device (e.g., a monitor or projector),speakers, a printer, a network card, and so forth. Display devices mayinclude display devices that provide visual information, thisinformation typically may be logically and/or physically organized as anarray of pixels. An interface controller may also be included that maycomprise any of a variety of known or future software programs forproviding input and output interfaces. For example, interfaces mayinclude what are generally referred to as “Graphical User Interfaces”(often referred to as GUI's) that provide one or more graphicalrepresentations to a user. Interfaces are typically enabled to acceptuser inputs using means of selection or input known to those of ordinaryskill in the related art. The interface may also be a touch screendevice. In the same or alternative embodiments, applications on acomputer may employ an interface that includes what are referred to as“command line interfaces” (often referred to as CLI's). CLI's typicallyprovide a text based interaction between an application and a user.Typically, command line interfaces present output and receive input aslines of text through display devices. For example, some implementationsmay include what are referred to as a “shell” such as Unix Shells knownto those of ordinary skill in the related art, or Microsoft WindowsPowershell that employs object-oriented type programming architecturessuch as the Microsoft .NET framework.

Those of ordinary skill in the related art will appreciate thatinterfaces may include one or more GUI's, CLI's or a combinationthereof. A processor may include a commercially available processor suchas a Celeron, Core, or Pentium processor made by Intel Corporation, aSPARC processor made by Sun Microsystems, an Athlon, Sempron, Phenom, orOpteron processor made by AMD Corporation, or it may be one of otherprocessors that are or will become available. Some embodiments of aprocessor may include what is referred to as multi-core processor and/orbe enabled to employ parallel processing technology in a single ormulti-core configuration. For example, a multi-core architecturetypically comprises two or more processor “execution cores”. In thepresent example, each execution core may perform as an independentprocessor that enables parallel execution of multiple threads. Inaddition, those of ordinary skill in the related will appreciate that aprocessor may be configured in what is generally referred to as 32 or 64bit architectures, or other architectural configurations now known orthat may be developed in the future.

A processor typically executes an operating system, which may be, forexample, a Windows type operating system from the Microsoft Corporation;the Mac OS X operating system from Apple Computer Corp.; a Unix orLinux-type operating system available from many vendors or what isreferred to as an open source; another or a future operating system; orsome combination thereof. An operating system interfaces with firmwareand hardware in a well-known manner, and facilitates the processor incoordinating and executing the functions of various computer programsthat may be written in a variety of programming languages. An operatingsystem, typically in cooperation with a processor, coordinates andexecutes functions of the other components of a computer. An operatingsystem also provides scheduling, input-output control, file and datamanagement, memory management, and communication control and relatedservices, all in accordance with known techniques.

System memory may include any of a variety of known or future memorystorage devices that can be used to store the desired information andthat can be accessed by a computer. Computer readable storage media mayinclude volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules, orother data. Examples include any commonly available random access memory(RAM), read-only memory (ROM), electronically erasable programmableread-only memory (EEPROM), digital versatile disks (DVD), magneticmedium, such as a resident hard disk or tape, an optical medium such asa read and write compact disc, or other memory storage device. Memorystorage devices may include any of a variety of known or future devices,including a compact disk drive, a tape drive, a removable hard diskdrive, USB or flash drive, or a diskette drive. Such types of memorystorage devices typically read from, and/or write to, a program storagemedium such as, respectively, a compact disk, magnetic tape, removablehard disk, USB or flash drive, or floppy diskette. Any of these programstorage media, or others now in use or that may later be developed, maybe considered a computer program product. As will be appreciated, theseprogram storage media typically store a computer software program and/ordata. Computer software programs, also called computer control logic,typically are stored in system memory and/or the program storage deviceused in conjunction with memory storage device. In some embodiments, acomputer program product is described comprising a computer usablemedium having control logic (computer software program, includingprogram code) stored therein. The control logic, when executed by aprocessor, causes the processor to perform functions described herein.In other embodiments, some functions are implemented primarily inhardware using, for example, a hardware state machine. Implementation ofthe hardware state machine so as to perform the functions describedherein will be apparent to those skilled in the relevant arts.Input-output controllers could include any of a variety of known devicesfor accepting and processing information from a user, whether a human ora machine, whether local or remote. Such devices include, for example,modem cards, wireless cards, network interface cards, sound cards, orother types of controllers for any of a variety of known input devices.Output controllers could include controllers for any of a variety ofknown display devices for presenting information to a user, whether ahuman or a machine, whether local or remote. In the presently describedembodiment, the functional elements of a computer communicate with eachother via a system bus. Some embodiments of a computer may communicatewith some functional elements using network or other types of remotecommunications. As will be evident to those skilled in the relevant art,an instrument control and/or a data processing application, ifimplemented in software, may be loaded into and executed from systemmemory and/or a memory storage device. All or portions of the instrumentcontrol and/or data processing applications may also reside in aread-only memory or similar device of the memory storage device, suchdevices not requiring that the instrument control and/or data processingapplications first be loaded through input-output controllers. It willbe understood by those skilled in the relevant art that the instrumentcontrol and/or data processing applications, or portions of it, may beloaded by a processor, in a known manner into system memory, or cachememory, or both, as advantageous for execution. Also, a computer mayinclude one or more library files, experiment data files, and aninternet client stored in system memory. For example, experiment datacould include data related to one or more experiments or assays, such asdetected signal values, or other values associated with one or moresequencing by synthesis (SBS) experiments or processes. Additionally, aninternet client may include an application enabled to access a remoteservice on another computer using a network and may for instancecomprise what are generally referred to as “Web Browsers”. In thepresent example, some commonly employed web browsers include MicrosoftInternet Explorer available from Microsoft Corporation, Mozilla Firefoxfrom the Mozilla Corporation, Safari from Apple Computer Corp., GoogleChrome from the Google Corporation, or other type of web browsercurrently known in the art or to be developed in the future. Also, inthe same or other embodiments an internet client may include, or couldbe an element of, specialized software applications enabled to accessremote information via a network such as a data processing applicationfor biological applications.

A network may include one or more of the many various types of networkswell known to those of ordinary skill in the art. For example, a networkmay include a local or wide area network that may employ what iscommonly referred to as a TCP/IP protocol suite to communicate. Anetwork may include a network comprising a worldwide system ofinterconnected computer networks that is commonly referred to as theinternet, or could also include various intranet architectures. Those ofordinary skill in the related arts will also appreciate that some usersin networked environments may prefer to employ what are generallyreferred to as “firewalls” (also sometimes referred to as PacketFilters, or Border Protection Devices) to control information traffic toand from hardware and/or software systems. For example, firewalls maycomprise hardware or software elements or some combination thereof andare typically designed to enforce security policies put in place byusers, such as for instance network administrators, etc.

Additional Embodiments

1. An image analysis system (220) configured for detecting a tissueregion in a digital image of a tissue slide, wherein a tissue sample ismounted on the tissue slide, the image analysis system comprising aprocessor (220) and a storage medium, the storage medium comprising aplurality of slide-type specific tissue detection routines (213, 214,215, 216) and a generic tissue detection routine (217), the imageanalysis system being configured for performing a method comprising:selecting one of the slide-type specific tissue detection routines;checking, before and/or while the selected slide-type specific tissuedetection routine is performed, if the selected slide-type specifictissue detection routine corresponds to the tissue slide type of theslide depicted in the digital image; if yes, automatically performingthe selected slide-type specific tissue detection routine for detectingthe tissue region in the digital image if no, automatically performingthe generic tissue detection routine for detecting the tissue region inthe digital image.

2. The image analysis system of embodiment 1, wherein the digital imageis a thumbnail image.

3. The image analysis system of any one of the previous embodiments,wherein the image analysis system comprises a user interface, whereinthe selection of the slide-type specific tissue detection routinecomprises receiving a user's selection of a tissue slide type via theuser interface; and selecting the one of the slide-type specific tissuedetection routines assigned the tissue slide type selected by the user.

4. The image analysis system of any one of the previous embodiments,wherein the generic tissue detection routine and one or more of theslide-type specific tissue detection routines comprise a sub-routine forgenerating a binary mask (M) that selectively masks non-tissue regions,the sub-routine comprising: computing a histogram (800) of a grayscaleversion of the digital image; extracting a plurality of intensitythresholds (rts) from the histogram; applying the plurality of intensitythresholds on the digital image for generating a plurality ofintermediate masks from the digital image; generating the binary mask(M) by combining all the intermediate masks.

5. The image analysis system of embodiment 4, the extraction of theplurality of intensity thresholds from the histogram comprising:identifying a max-grayscale-index (mgs), the max-grayscale-index beingthe maximum grayscale intensity value observed in the histogram;identifying a max-histogram-location (mhl), the max-histogram-locationbeing the grayscale intensity value having the highest occurrencefrequency in the histogram; computing a rightgap by subtracting themaximum-histogram-location (mhl)from the max-grayscale-index (mgs);computing a glass-left-cutoff according toglass-left-cutoff=max-histogram-location(mhl)-rightgap; performing theextraction of the plurality of thresholds such that the lowest one(dco[0]) of the thresholds is an intensity value equal to or larger thanglass_left_cutoff−rightgap and such that the highest one of thethresholds is equal to or smaller than glass_left_cutoff+rightgap.

6. The image analysis system of embodiment 5, the extraction of theplurality of intensity thresholds from the histogram comprising:computing an interval-range-gap(irg) according to irg=max(1,round(rightgap/constant)), wherein constant is a predefined numericalvalue between 2 and 20, preferentially between 3 and 8; computing amax_rightgap value according to max_rightgap=min(rightgap−1,round(rightgap * gap_cutoff_fraction), wherein the gap_cutoff_fractionis a predefined value in the range of 0.5-0.95, preferentially 0.7-0.8;computing a number_interval_terms according tonumber_interval_terms=((max_rightgap+rightgap)/interval_range_gap);creating the plurality of threshold such that their number is identicalto the computed number_interval_terms.

7. The image analysis system of embodiments 4, 5 or 6, the applicationof the plurality of intensity thresholds (rts) on the digital image forgenerating a plurality of intermediate masks comprising: a) creating afirst intermediate mask by masking all pixels in the digital image whosegrayscale value is below a first one of the plurality of thresholds; b)creating a second intermediate mask by masking all pixels in the digitalimage whose grayscale value is below a second one of the plurality ofthresholds; c) performing a connected-component-analysis on the unmaskedpixels of the first mask for identifying first blobs; d) performing aconnected-component-analysis on the unmasked pixels of the second maskfor identifying second blobs; e) merging the first and second mask intoa merged mask, the merging comprising: masking all first blobs whosesize is below an absolute-minimum-area and masking all first blobs whosesize is above the absolute-minimum-area but below aconditional-minimum-area and which lack a first blob in a predefinedneighborhood area around said first blob; after having performed themasking of one or more of the first blobs, unifying the unmasked regionsof the first and second mask to generate the unmasked regions of themerged mask, all other regions of the merged mask being masked regions;f) using the merged mask as a new first intermediate mask, selecting athird one of the threshold for computing a new second intermediate maskfrom the third threshold, repeating steps c)-e) until each of theplurality of thresholds was selected, and outputting the finallygenerated merged mask as the merged binary mask (M).

8. The image analysis system of embodiment 7, the absolute-minimum-areabeing in the range of 100%-200% of an expected size of a blobrepresenting a tissue cell, the conditional-minimum-area being in therange of 30%-70% of an expected size of a blob representing a tissuecell.

9. The image analysis system of any one of the previous embodiments,wherein the slide-type specific tissue detection routines comprise aTMA-slide-routine, a TMA-slide routine being configured for detectingthe tissue region in a tissue micro array (TMA) slide, the TMA-slideroutine being configured for: generating a binary mask (M) by performingthe sub-routine of any one of embodiments 4-8; applying the binary mask(M) on the digital image and selectively analyzing unmasked regions inthe digital image for detecting a grid of tissue regions.

10. The image analysis system of embodiment 9, wherein the selectedslide-type specific tissue detection routine is a TMA-slide routine andthe image analysis system is further configured for: if the grid oftissue regions is not detected in the unmasked regions of the digitalimage, determining that the selected TMA-slide tissue detection routinedoes not correspond to the tissue slide type of the tissue slidedepicted in the digital image; and terminating execution of theTMA-slide tissue detection routine and starting execution of the generictissue detection routine.

11. The image analysis system of any one of the previous embodiments1-8, wherein the slide-type specific tissue detection routines comprisea control-slide-routine, a control-slide routine being configured fordetecting the tissue region in a control slide, the control-slideroutine being configured for: generating a binary mask (M) by performingthe sub-routine of any one of embodiments 4-8; applying the binary mask(M) on the digital image; and selectively analyzing unmasked regions inthe digital image for detecting a number of tissue regions positioned ina straight line.

12. The image analysis system of embodiment 11, wherein the selectedslide-type specific tissue detection routine is a control-slide routineand the image analysis system is further configured for: if the numberof tissue regions in a straight line are not detected in the unmaskedregions of the digital image, determining that the selectedcontrol-slide tissue detection routine does not correspond to the tissueslide type of the tissue slide depicted in the digital image; andterminating execution of the control-slide tissue detection routine andstarting execution of the generic tissue detection routine.

13. The image analysis system of any one of the previous embodiments1-8, wherein the slide-type specific tissue detection routines comprisea cell-line-slide-routine, a cell-line-slide-routine being configuredfor detecting the tissue region in a cell-line-slide, a cell-line-slidebeing a tissue slide comprising a single, disc-shape tissue sample, thecell-line-slide routine being configured for: computing a grayscaleimage as a derivative of the digital image; computing a gradientmagnitude image as a derivative of the grayscale image; computing ahistogram (800) of the digital image; extracting a plurality ofintensity thresholds (rts) from the histogram; applying a plurality ofintensity threshold values on the digital image for generating aplurality of intermediate masks from the digital image; generating abinary mask (M) by combining all the intermediate masks; applying thebinary mask (M) on the digital image and the gradient magnitude image;and performing radial symmetry based voting selectively on the unmaskedregion in the gradient magnitude image for detecting a center and aradius of a single tissue region.

14. The image analysis system of embodiment 13, wherein the selectedslide-type specific tissue detection routine is a cell-line-slideroutine and the image analysis system is further configured for: if thesingle tissue region was not detected or if the detected radius or thedetected location of the center deviates from a radius or centerlocation expected for the cell-line-slide type, determining that theselected cell-line-slide tissue detection routine does not correspond tothe tissue slide type of the tissue slide depicted in the digital image;and terminating execution of the cell-line-slide tissue detectionroutine and starting execution of the generic tissue detection routine.

15. The image analysis system of any one of the previous embodiments1-8, wherein the slide-type specific tissue detection routines comprisea smear-slide-routine, a smear-slide routine being configured fordetecting the tissue region in a smear slide, a smear slide being aslide type where on which the tissue region is diffused throughout theslide, the smear-slide routine being configured for: computing aluminance image as a derivative of the digital image, the luminanceimage being an L-channel image of a LUV color space representation ofthe digital image; computing a chrominance image as a derivative fromthe digital image, the chrominance image being a derivative of theU,V-channels of the LUV color space representation of the digital image;performing hysteresis thresholding on an inverted version of theluminance image for generating a luminance-based mask; performinghysteresis thresholding on an inverted version of the chrominance imagefor generating a chrominance-based mask; combining the luminance-basedmask and the chrominance-based mask to obtain a binary mask (M);applying the binary mask on the digital image; and selectively analyzingunmasked regions in the digital image for detecting the diffused tissueregion.

16. The image analysis system of embodiment 15, wherein performing thehysteresis thresholding comprises: determining, for each set of adjacentpixels whose pixel value meets a lower and an upper threshold appliedduring the hysteresis thresholding, if the set of pixels covers at leastan empirically determined area; if yes, generating the binary mask suchthat said set of pixels is unmasked; if no, generating the binary masksuch that said set of pixels is masked.

17. The image analysis system of any one of the previous embodiments,wherein the tissue slide is a glass slide and the tissue detectionroutines are configured for generating masks that selectively mask glassregions, wherein tissue regions are unmasked regions in said masks.

18. The image analysis system according to any one of the previousembodiments 4-17, further comprising using size and distance constraintsto detect and discard smaller regions from intermediate masks whencombining the intermediate masks to generate the binary mask.

19. The image analysis system according to any one of the previousembodiments, the generic and one or more of thetissue-slide-type-specific tissue detection routines being furtherconfigured for: generating a weighted image from the digital image, eachpixel in the weighted image having assigned a weight in a value rangecomprising more than two different values, the weight indicating alikelihood that said pixel belongs to a tissue region as compared tobelonging to glass.

20. The image analysis system of embodiment 19, the generation of theweighted image comprising: computing a luminance image as a derivativeof the digital image, the luminance image being an L-channel image of aLUV color space representation of the digital image, or accessing aluminance image having been computed already for generating a binarymask (M); computing a chrominance image as a derivative from the digitalimage, the chrominance image being a derivative of the U,V-channels ofthe LUV color space representation of the digital image, or accessing achrominance image having been computed already for generating the binarymask (M); masking the luminance and/or the chrominance with a binarymask (M), the binary mask selectively masking non-tissue regions;generating a weighted image (SW) as a derivative of the luminance and/orthe chrominance image, wherein each pixel in the weighted image hasassigned a value that positively correlates with the chrominance of acorresponding pixel in the luminance image and/or that positivelycorrelates with the darkness of a corresponding pixel in the luminanceimage.

21. The image analysis system according to any one of the previousembodiments, the image analysis system comprising a graphical userinterface configured for: displaying a selectable GUI element (219), theGUI element enabling a user to select one of the slide-type specific orthe generic tissue detection routines; and/or displaying the detectedtissue regions, wherein the color and/or brightness of the pixels of thedisplayed tissue regions depend on the weights assigned to each of thetissue region pixels; and/or enabling a user to add extra focus pointson regions of the displayed tissue regions, the adding of the extrafocus points triggering a scanner to scan regions indicated by the focuspoints in more detail.

22. An image analysis method for detecting a tissue region in a digitalimage of a tissue slide, wherein a tissue sample is mounted on thetissue slide, the method being implemented in an image analysis systemcomprising a processor and a storage medium, the method comprising:selecting one of the slide-type specific tissue detection routines;checking, before and/or while the selected slide-type specific tissuedetection routine is performed, if the selected slide-type specifictissue detection routine corresponds to the tissue slide type of theslide depicted in the digital image; if yes, automatically performingthe selected slide-type specific tissue detection routine for detectingthe tissue region in the digital image if no, automatically performingthe generic tissue detection routine for detecting the tissue region inthe digital image.

23. A storage medium comprising instructions interpretable by aprocessor (220) of an image analysis system, the instructions whenexecuted by the processor cause the image analysis system to perform themethod according to embodiment 22.

24. A system for detecting an area-of-interest (AOI) on a thumbnailimage of a tissue slide, the system comprising: a processor (220); and amemory coupled to the processor, the memory to store computer-readableinstructions that, when executed by the processor, cause the processorto perform operations comprising: receiving a thumbnail image from acamera coupled to the processor; receiving, via a user interface, aselection of a thumbnail image type; and determining an area of interest(AOI) from the thumbnail image using one of a plurality of AOI detectionmethods corresponding to the thumbnail image type.

25. The system of embodiment 24, wherein the thumbnail image type isselected from among a plurality of thumbnail image type choices providedon the user interface.

26. The system of embodiment 24, wherein the operations further comprisedetermining that the thumbnail image type is incorrectly input, andusing another one of the plurality of AOI detection methods.

27. The system of embodiment 26, wherein the thumbnail image type is ofa tissue micro array (TMA) slide, and wherein the operations furthercomprise attempting to detect a grid of cores based on a binary maskgenerated using a range of threshold values computed from an imagehistogram of the thumbnail image.

28. The system of embodiment 27, wherein if the grid of cores is notdetected in the thumbnail image, then the input is determined to beincorrect and a generic method for AOI detection is used.

29. The system of embodiment 26, wherein the thumbnail image type is ofa control slide, and wherein the operations further comprise attemptingto detect a number of cores positioned in a straight line in thethumbnail image.

30. The system of embodiment 29, wherein if the number of cores in astraight line are not detected, then the input is determined to beincorrect and a generic method for AOI detection is used.

31. The system of embodiment 24, wherein the thumbnail image type is ofa ThinPrep(RTM) slide or equivalent, and wherein the operations furthercomprise detecting the center and radius of the single tissue core whileperforming radial symmetry based voting on a gradient magnitude of adown-sampled version of an enhanced grayscale version of the thumbnailimage.

32. The system of embodiment 24, wherein the thumbnail image type isthat of a smear slide, and wherein the operations further comprisedetermining lower and upper thresholds for a luminance and a chrominanceimage derived from the thumbnail image, and then performing hysteresisthresholding on the luminance and chrominance images of the thumbnailimage to obtain a final effective binary mask.

33. The system of embodiment 32, wherein the hysteresis thresholding isperformed based on an assumption that a smear tissue sample is spreadall over the slide, and based on an empirically determined areaconstraint.

34. The system of embodiment 24, wherein the thumbnail image type isthat of a generic slide, and wherein the operations further compriseobtaining an initial estimate of the tissue region as compared to glass,by using a range of thresholds computed based on a histogram of thethumbnail image.

35. The system of embodiment 34, wherein, to obtain a more precise AOIregion, the operations further comprise using size and distanceconstraints to detect and discard smaller regions from the AOI.

36. The system of embodiment 24, wherein the operations further comprisegenerating a soft-weighted image depicting the AOI tissue probabilityimage.

37. The system of embodiment 35, wherein weights are assigned to pixelsin the soft-weighted image based on a likelihood that they represent atissue region as compared to belonging to glass.

38. The system of embodiment 36, wherein the generating of thesoft-weighted image uses the thumbnail image as well as a binary mask asan input, wherein the soft-weighted image is zero valued at all pixelsexcept the pixels which are non-zero inside the binary mask, and thesenon-zero pixels get assigned higher or lower values depending on whetherthey are more or less likely to belong to tissue as compared tobelonging to glass.

39. The system of embodiment 37, wherein the binary mask is generated bysaid one or another of the plurality of AOI detection methods.

40. The system of embodiment 35, wherein the operations further comprisedisplaying the soft-weighted AOI image on a display.

41. The system of embodiment 39, wherein the operations further compriseproviding a user interface enabling a user to add extra focus points onindicated regions of the displayed soft-weighted AOI image, the extrafocus points enabling a scanner to scan the indicated regions in moredetail.

42. A system for detecting an area-of-interest (AOI) on a thumbnailimage of a tissue slide, comprising: a processor; and a memory coupledto the processor, the memory to store computer-readable instructionsthat, when executed by the processor, cause the processor to performoperations comprising: determining an area of interest (AOI) from athumbnail image using one of a plurality of AOI detection methodsdepending on the thumbnail image type; and outputting a soft-weightedimage depicting the detected AOI; wherein the thumbnail image type isinput by a user via a user interface and represents one of a ThinPrep(RTM) slide, a tissue microarray slide, a control slide, a smear slide,or a generic slide.

43. A system for detecting an area-of-interest (AOI) on a thumbnailimage of a tissue slide, comprising: a processor; and a memory coupledto the processor, the memory to store computer-readable instructionsthat, when executed by the processor, cause the processor to performoperations comprising: receiving an input comprising a thumbnail imageand a thumbnail image type; determining an area of interest (AOI) fromthe thumbnail image using one of a plurality of AOI detection methodsdepending on the thumbnail image type, the plurality of AOI detectionmethods including a ThinPrep (RTM) method, a tissue microarray method, acontrol method, a smear method, or a generic method; and upondetermining that the thumbnail image type is incorrectly input, usingthe generic method

The foregoing disclosure of the exemplary embodiments of the presentsubject disclosure has been presented for purposes of illustration anddescription. It is not intended to be exhaustive or to limit the subjectdisclosure to the precise forms disclosed. Many variations andmodifications of the embodiments described herein will be apparent toone of ordinary skill in the art in light of the above disclosure. Thescope of the subject disclosure is to be defined only by the claimsappended hereto, and by their equivalents.

Further, in describing representative embodiments of the present subjectdisclosure, the specification may have presented the method and/orprocess of the present subject disclosure as a particular sequence ofsteps. However, to the extent that the method or process does not relyon the particular order of steps set forth herein, the method or processshould not be limited to the particular sequence of steps described. Asone of ordinary skill in the art would appreciate, other sequences ofsteps may be possible. Therefore, the particular order of the steps setforth in the specification should not be construed as limitations on theclaims. In addition, the claims directed to the method and/or process ofthe present subject disclosure should not be limited to the performanceof their steps in the order written, and one skilled in the art canreadily appreciate that the sequences may be varied and still remainwithin the spirit and scope of the present subject disclosure.

What is claimed is:
 1. An image analysis system configured for detectinga tissue region in a digital image of a tissue slide, wherein a tissuesample is mounted on the tissue slide, the image analysis systemcomprising a processor and a storage medium, the storage mediumcomprising a plurality of slide-type specific tissue detection routinesand a generic tissue detection routine, the image analysis system beingconfigured for performing a method comprising: selecting one of theslide-type specific tissue detection routines; checking, before and/orwhile the selected slide-type specific tissue detection routine isperformed, if the selected slide-type specific tissue detection routinecorresponds to the tissue slide type of the slide depicted in thedigital image; if yes, automatically performing the selected slide-typespecific tissue detection routine for detecting the tissue region in thedigital image if no, automatically performing the generic tissuedetection routine for detecting the tissue region in the digital image.2. The image analysis system of claim 1, wherein the digital image is athumbnail image.
 3. The image analysis system of claim 1, wherein theimage analysis system comprises a user interface, wherein the selectionof the slide-type specific tissue detection routine comprises receivinga user's selection of a tissue slide type via the user interface; andselecting the one of the slide-type specific tissue detection routinesassigned the tissue slide type selected by the user.
 4. The imageanalysis system of claim 1, wherein the generic tissue detection routineand one or more of the slide-type specific tissue detection routinescomprise a sub-routine for generating a binary mask (M) that selectivelymasks non-tissue regions, the sub-routine comprising: computing ahistogram of a grayscale version of the digital image; extracting aplurality of intensity thresholds (rts) from the histogram; applying theplurality of intensity thresholds on the digital image for generating aplurality of intermediate masks from the digital image; generating thebinary mask (M) by combining all the intermediate masks.
 5. The imageanalysis system of claim 4, the extraction of the plurality of intensitythresholds from the histogram comprising: identifying amax-grayscale-index (mgs), the max-grayscale-index being the maximumgrayscale intensity value observed in the histogram; identifying amax-histogram-location (mhl), the max-histogram-location being thegrayscale intensity value having the highest occurrence frequency in thehistogram; computing a rightgap by subtracting themaximum-histogram-location (mhl)from the max-grayscale-index (mgs);computing a glass-left-cutoff according toglass-left-cutoff=max-histogram-location(mhl)-rightgap; and performingthe extraction of the plurality of thresholds such that the lowest one(dco[0]) of the thresholds is an intensity value equal to or larger thanglass_left_cutoff−rightgap and such that the highest one of thethresholds is equal to or smaller than glass_left_cutoff+rightgap. 6.The image analysis system of claim 5, the extraction of the plurality ofintensity thresholds from the histogram comprising: computing aninterval-range-gap(irg) according to irg=max(1,round(rightgap/constant)), wherein constant is a predefined numericalvalue between 2 and 20, preferentially between 3 and 8; computing amax_rightgap value according to max_rightgap=min(rightgap−1,round(rightgap * gap_cutoff_fraction), wherein the gap_cutoff_fractionis a predefined value in the range of 0.5-0.95, preferentially 0.7-0.8;computing a number_interval_terms according tonumber_interval_terms=((max_rightgap+rightgap)/interval_range_gap); andcreating the plurality of threshold such that their number is identicalto the computed number_interval_terms.
 7. The image analysis system ofclaim 4, the application of the plurality of intensity thresholds (rts)on the digital image for generating a plurality of intermediate maskscomprising: a) creating a first intermediate mask by masking all pixelsin the digital image whose grayscale value is below a first one of theplurality of thresholds; b) creating a second intermediate mask bymasking all pixels in the digital image whose grayscale value is below asecond one of the plurality of thresholds; c) performing aconnected-component-analysis on the unmasked pixels of the first maskfor identifying first blobs; d) performing aconnected-component-analysis on the unmasked pixels of the second maskfor identifying second blobs; e) merging the first and second mask intoa merged mask, the merging comprising: masking all first blobs whosesize is below an absolute-minimum-area and masking all first blobs whosesize is above the absolute-minimum-area but below aconditional-minimum-area and which lack a first blob in a predefinedneighborhood area around said first blob; after having performed themasking of one or more of the first blobs, unifying the unmasked regionsof the first and second mask to generate the unmasked regions of themerged mask, all other regions of the merged mask being masked regions;and f) using the merged mask as a new first intermediate mask, selectinga third one of the threshold for computing a new second intermediatemask from the third threshold, repeating steps c)-e) until each of theplurality of thresholds was selected, and outputting the finallygenerated merged mask as the merged binary mask (M).
 8. The imageanalysis system of claim 7, the absolute-minimum-area being in the rangeof 100%-200% of an expected size of a blob representing a tissue cell,the conditional-minimum-area being in the range of 30%-70% of anexpected size of a blob representing a tissue cell.
 9. The imageanalysis system of claim 1, wherein the slide-type specific tissuedetection routines comprise a TMA-slide-routine, a TMA-slide routinebeing configured for detecting the tissue region in a tissue micro array(TMA) slide, the TMA-slide routine being configured for: generating abinary mask (M) by performing the sub-routine of claim 4; applying thebinary mask (M) on the digital image and selectively analyzing unmaskedregions in the digital image for detecting a grid of tissue regions, andif the grid of tissue regions is not detected in the unmasked regions ofthe digital image, determining that the selected TMA-slide tissuedetection routine does not correspond to the tissue slide type of thetissue slide depicted in the digital image; and terminating execution ofthe TMA-slide tissue detection routine and starting execution of thegeneric tissue detection routine.
 10. The image analysis system of claim1, wherein the slide-type specific tissue detection routines comprise acontrol-slide-routine, a control-slide routine being configured fordetecting the tissue region in a control slide, the control-slideroutine being configured for: generating a binary mask (M) by performingthe sub-routine of any one of claims 4-8; applying the binary mask (M)on the digital image; and selectively analyzing unmasked regions in thedigital image for detecting a number of tissue regions positioned in astraight line, and if the number of tissue regions in a straight lineare not detected in the unmasked regions of the digital image,determining that the selected control-slide tissue detection routinedoes not correspond to the tissue slide type of the tissue slidedepicted in the digital image; and terminating execution of thecontrol-slide tissue detection routine and starting execution of thegeneric tissue detection routine.
 11. The image analysis system of claim4, wherein the slide-type specific tissue detection routines comprise acell-line-slide-routine, a cell-line-slide-routine being configured fordetecting the tissue region in a cell-line-slide, a cell-line-slidebeing a tissue slide comprising a single, disc-shape tissue sample, thecell-line-slide routine being configured for: computing a grayscaleimage as a derivative of the digital image; computing a gradientmagnitude image as a derivative of the grayscale image; computing ahistogram of the digital image; extracting a plurality of intensitythresholds (rts) from the histogram; applying a plurality of intensitythreshold values on the digital image for generating a plurality ofintermediate masks from the digital image; generating a binary mask (M)by combining all the intermediate masks; applying the binary mask (M) onthe digital image and the gradient magnitude image; and performingradial symmetry based voting selectively on the unmasked region in thegradient magnitude image for detecting a center and a radius of a singletissue region, and if the single tissue region was not detected or ifthe detected radius or the detected location of the center deviates froma radius or center location expected for the cell-line-slide type,determining that the selected cell-line-slide tissue detection routinedoes not correspond to the tissue slide type of the tissue slidedepicted in the digital image; and terminating execution of thecell-line-slide tissue detection routine and starting execution of thegeneric tissue detection routine.
 12. The image analysis system of claim1, wherein the slide-type specific tissue detection routines comprise asmear-slide-routine, a smear-slide routine being configured fordetecting the tissue region in a smear slide, a smear slide being aslide type where on which the tissue region is diffused throughout theslide, the smear-slide routine being configured for: computing aluminance image as a derivative of the digital image, the luminanceimage being an L-channel image of a LUV color space representation ofthe digital image; computing a chrominance image as a derivative fromthe digital image, the chrominance image being a derivative of theU,V-channels of the LUV color space representation of the digital image;performing hysteresis thresholding on an inverted version of theluminance image for generating a luminance-based mask; performinghysteresis thresholding on an inverted version of the chrominance imagefor generating a chrominance-based mask; combining the luminance-basedmask and the chrominance-based mask to obtain a binary mask (M);applying the binary mask on the digital image; and selectively analyzingunmasked regions in the digital image for detecting the diffused tissueregion, wherein performing the hysteresis thresholding comprises:determining, for each set of adjacent pixels whose pixel value meets alower and an upper threshold applied during the hysteresis thresholding,if the set of pixels covers at least an empirically determined area; andif yes, generating the binary mask such that said set of pixels isunmasked; if no, generating the binary mask such that said set of pixelsis masked.
 13. The image analysis system of claim 1, wherein the tissueslide is a glass slide and the tissue detection routines are configuredfor generating masks that selectively mask glass regions, wherein tissueregions are unmasked regions in said masks.
 14. The image analysissystem of claim 4, further comprising using size and distanceconstraints to detect and discard smaller regions from intermediatemasks when combining the intermediate masks to generate the binary mask.15. The image analysis system of claim 1, the generic and one or more ofthe tissue-slide-type-specific tissue detection routines being furtherconfigured for: generating a weighted image from the digital image, eachpixel in the weighted image having assigned a weight in a value rangecomprising more than two different values, the weight indicating alikelihood that said pixel belongs to a tissue region as compared tobelonging to glass, the generation of the weighted image comprising:computing a luminance image as a derivative of the digital image, theluminance image being an L-channel image of a LUV color spacerepresentation of the digital image, or accessing a luminance imagehaving been computed already for generating a binary mask (M); computinga chrominance image as a derivative from the digital image, thechrominance image being a derivative of the U,V-channels of the LUVcolor space representation of the digital image, or accessing achrominance image having been computed already for generating the binarymask (M); masking the luminance and/or the chrominance with a binarymask (M), the binary mask selectively masking non-tissue regions;generating a weighted image (SW) as a derivative of the luminance and/orthe chrominance image, wherein each pixel in the weighted image hasassigned a value that positively correlates with the chrominance of acorresponding pixel in the luminance image and/or that positivelycorrelates with the darkness of a corresponding pixel in the luminanceimage.
 16. The image analysis system of claim 1, the image analysissystem comprising a graphical user interface configured for: displayinga selectable GUI element (219), the GUI element enabling a user toselect one of the slide-type specific or the generic tissue detectionroutines; and/or displaying the detected tissue regions, wherein thecolor and/or brightness of the pixels of the displayed tissue regionsdepend on the weights assigned to each of the tissue region pixels;and/or enabling a user to add extra focus points on regions of thedisplayed tissue regions, the adding of the extra focus pointstriggering a scanner to scan regions indicated by the focus points inmore detail.
 17. An image analysis method for detecting a tissue regionin a digital image of a tissue slide, wherein a tissue sample is mountedon the tissue slide, the method being implemented in an image analysissystem comprising a processor and a storage medium, the methodcomprising: selecting one of the slide-type specific tissue detectionroutines; checking, before and/or while the selected slide-type specifictissue detection routine is performed, if the selected slide-typespecific tissue detection routine corresponds to the tissue slide typeof the slide depicted in the digital image; if yes, automaticallyperforming the selected slide-type specific tissue detection routine fordetecting the tissue region in the digital image if no, automaticallyperforming the generic tissue detection routine for detecting the tissueregion in the digital image.
 18. A storage medium comprisinginstructions interpretable by a processor of an image analysis system,the instructions when executed by the processor cause the image analysissystem to perform the method according to claim
 17. 19. A system fordetecting an area-of-interest (AOI) on a thumbnail image of a tissueslide, the system comprising: a processor; and a memory coupled to theprocessor, the memory to store computer-readable instructions that, whenexecuted by the processor, cause the processor to perform operationscomprising: receiving a thumbnail image from a camera coupled to theprocessor, wherein the thumbnail image type is selected from among aplurality of thumbnail image type choices provided on the userinterface; receiving, via a user interface, a selection of a thumbnailimage type; determining an area of interest (AOI) from the thumbnailimage using one of a plurality of AOI detection methods corresponding tothe thumbnail image type; determining that the thumbnail image type isincorrectly input; and using another one of the plurality of AOIdetection methods.
 20. The system of claim 19, wherein the thumbnailimage type is of a tissue micro array (TMA) slide, and wherein theoperations further comprise attempting to detect a grid of cores basedon a binary mask generated using a range of threshold values computedfrom an image histogram of the thumbnail image, and wherein if the gridof cores is not detected in the thumbnail image, then the input isdetermined to be incorrect and a generic method for AOI detection isused.
 21. The system of claim 19, wherein the thumbnail image type is ofa control slide, and wherein the operations further comprise attemptingto detect a number of cores positioned in a straight line in thethumbnail image, and wherein if the number of cores in a straight lineare not detected, then the input is determined to be incorrect and ageneric method for AOI detection is used.
 22. The system of claim 19,wherein the thumbnail image type is of a ThinPrep(RTM) slide orequivalent, and wherein the operations further comprise detecting thecenter and radius of the single tissue core while performing radialsymmetry based voting on a gradient magnitude of a down-sampled versionof an enhanced grayscale version of the thumbnail image.
 23. The systemof claim 19, wherein the thumbnail image type is that of a smear slide,and wherein the operations further comprise determining lower and upperthresholds for a luminance and a chrominance image derived from thethumbnail image, and then performing hysteresis thresholding on theluminance and chrominance images of the thumbnail image to obtain afinal effective binary mask, and wherein the hysteresis thresholding isperformed based on an assumption that a smear tissue sample is spreadall over the slide, and based on an empirically determined areaconstraint.
 24. The system of claim 19, wherein the thumbnail image typeis that of a generic slide, and wherein the operations further compriseobtaining an initial estimate of the tissue region as compared to glass,by using a range of thresholds computed based on a histogram of thethumbnail image, and to obtain a more precise AOI region, the operationsfurther comprise using size and distance constraints to detect anddiscard smaller regions from the AOI.
 25. The system of claim 19,wherein the operations further comprise generating a soft-weighted imagedepicting the AOI tissue probability image, and wherein weights areassigned to pixels in the soft-weighted image based on a likelihood thatthey represent a tissue region as compared to belonging to glass, andwherein the generating of the soft-weighted image uses the thumbnailimage as well as a binary mask as an input, wherein the soft-weightedimage is zero valued at all pixels except the pixels which are non-zeroinside the binary mask, and these non-zero pixels get assigned higher orlower values depending on whether they are more or less likely to belongto tissue as compared to belonging to glass.